Instructions to use Heitechsoft/FalconAlpaca-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Heitechsoft/FalconAlpaca-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Heitechsoft/FalconAlpaca-7B", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Heitechsoft/FalconAlpaca-7B", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Heitechsoft/FalconAlpaca-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Heitechsoft/FalconAlpaca-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Heitechsoft/FalconAlpaca-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Heitechsoft/FalconAlpaca-7B

SGLang

How to use Heitechsoft/FalconAlpaca-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Heitechsoft/FalconAlpaca-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Heitechsoft/FalconAlpaca-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Heitechsoft/FalconAlpaca-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Heitechsoft/FalconAlpaca-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Heitechsoft/FalconAlpaca-7B with Docker Model Runner:
```
docker model run hf.co/Heitechsoft/FalconAlpaca-7B
```

Model Card for FalconAlpaca

FalconAlpaca is Falcon-7B trained on the Stanford Alpaca Dataset

Model Details

This model was an attempt to influence the learned outputs of Falcon-7B to adapt the outputs to become more information-rich and focused. Trained using Lit GPT, the model took 2 hours to train on 1 4xA6000 node.

Model Description

License: [Apache 2.0]
Finetuned from model : Falcon-7B

Model Sources

Stanford Alpaca Dataset

Out-of-Scope Use

This model is not intended for anything but testing purposes. There have been no attempts to control/remove bias, toxicity, or any other form of potentially dangerous or harmful messages.

Bias, Risks, and Limitations

No effort was made to remove any wrong or harmful information from Falcon-7B or the Alpaca dataset. Any risks and limitations from either of those datasets/models carry over to this project as well.

How to Get Started with the Model

Download and install libraries for Lit GPT

python generate/adapter_v2.py \
    --adapter_path path/to/model/lit_model_adapter_finetuned.pth \
    --checkpoint_dir path/to/model \
    --prompt "What temperature should I cook pork at to ensure it is safe?"

This uses around 14GB of VRAM. If you need to use less VRAM, you can add the parameters

--quantize llm.int8

--quantize gptq.int4

Training Data

Stanford Alpaca Dataset

Training Hyperparameters

The defaults were as follows

learning_rate = 9e-3
batch_size = 32
micro_batch_size = 2
gradient_accumulation_iters = 16
epoch_size = 50000
num_epochs = 5
max_iters = 125000
weight_decay = 0.02
warmup_iters = 50000

More Information

HeitechSoft

Downloads last month: 5