반응형
맥북 ARM 계열 CPU(M1 등등)에서 tokenizers 라이브러리가 정상적으로 설치가 안되는 오류가 발생한다.
sentence_transformers 등 여러 라이브러리랑도 엮여있어서 필수로 필요한 라이브러리와 다름없는데, 설치가 안될 경우 아래 명령어를 따라 치면 정상적으로 해결된다.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
git clone https://github.com/huggingface/tokenizers
pip install setuptools_rust
pip install tokenizers
아래는 내가 났던 Tokenizers 라이브러리 설치 오류다.
Building wheels for collected packages: tokenizers
Building wheel for tokenizers (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [51 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-11.0-arm64-cpython-38
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers
copying py_src/tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
copying py_src/tokenizers/models/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
copying py_src/tokenizers/decoders/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
copying py_src/tokenizers/normalizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
copying py_src/tokenizers/processors/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
copying py_src/tokenizers/trainers/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/implementations
creating build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
copying py_src/tokenizers/tools/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
copying py_src/tokenizers/tools/visualizer.py -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
copying py_src/tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers
copying py_src/tokenizers/models/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/models
copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/decoders
copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/normalizers
copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/pre_tokenizers
copying py_src/tokenizers/processors/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/processors
copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/trainers
copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.macosx-11.0-arm64-cpython-38/tokenizers/tools
running build_ext
running build_rust
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
반응형
'프로그래밍 > 파이썬(Python)' 카테고리의 다른 글
파이썬 tkinter를 이용한 숫자 맞추기 게임 배포 (0) | 2020.12.15 |
---|---|
Pyinstaller을 이용한 파이썬 -> exe화 시키기! (3.7까지 지원) (0) | 2020.02.15 |
Pyinstaller를 이용한 파이썬파일을 윈도우실행파일로 만들기!(Python 3.x버전 기준) (0) | 2018.02.20 |
파이썬으로 윈도우 창 띄우기! #7 콤보선택박스를 넣어보자(feat.tkinter) (0) | 2018.02.16 |
파이썬으로 윈도우 창 띄우기! #6 라디오 선택 버튼을 넣어보자(feat.tkinter) (0) | 2018.02.15 |