Oracle Cloud Infrastructure Documentation

Fine-Tuning Hyperparameters in Generative AI

meta.llama-3.3-70b-instruct is the pretrained base model currently available for fine-tuning in OCI Generative AI. This page lists its supported hyperparameters, valid ranges, and default values.

Note

The standard meta.llama-3.3-70b-instruct variant is available for fine-tuning. The meta.llama-3.3-70b-instruct-fp8-dynamic variant isn't available for fine-tuning. See Meta Llama 3.3 (70B) and Models by Region.

Before fine-tuning, review the training data requirements. For the complete process, see Fine-Tuning Workflow.

Tip

Start fine-tuning with the default hyperparameter values. After the custom model is created, on the model's details page, under Model Performance, review the values for accuracy and loss. If the results don't meet your needs, create another custom model with a larger dataset or different hyperparameters.

meta.llama-3.3-70b-instruct

The following table outlines the hyperparameters OCI Generative AI uses to train a meta.llama-3.3-70b-instruct base model with the LoRA method.

Hyperparameter Description Valid Range Default Value
Total training epochs The number of times the training iterates through the entire training dataset. For example, 1 epoch means that the model is trained by using the entire training dataset one time.

1 or a higher integer

 3
Learning rate The speed at which the model weights are updated against the error gradient. A number between 0 and 1.0 0.0002
Training batch size The number of samples in a mini batch to go through before updating the model's parameters. An integer between 8 and 16 8
Early stopping patience Defines the number of grace periods to continue the evaluation cycle, after the early stopping threshold is triggered. Training stops if the loss metric doesn't improve beyond the early stopping threshold for this many times of evaluation. 0 to disable and 1 or a higher integer to add a grace period 15
Early stopping threshold Loss improves when it decreases in the next training cycle. If loss doesn't improve enough, you can stop the training. Define the minimum evaluation loss improvement that should trigger the early stopping counter. If loss doesn't improve beyond the minimum value during the patience period, training stops. Otherwise, training continues and the counter resets. 0 or a positive number 0.0001
Log model metrics interval in steps The number of steps per logging. Model metrics such as training loss and learning rate are logged. If the training loss is not decreasing as expected, review the training data or learning rate. Preset to 10 10
LoRA r (for LoRA method only) The attention dimension (rank) of the update matrices. A lower rank results in smaller update matrices with fewer trainable parameters. An integer between 1 and 64 8
LoRA alpha (for LoRA method only) The alpha parameter for LoRA scaling. The LoRA weight matrices are scaled by dividing LoRA alpha by LoRA r. The alpha parameter defines the LoRA weights, which are a smaller number of new weights and are the only weights that are trained in the model. An integer between 1 and 128 8
LoRA dropout (for LoRA method only) The dropout probability for neurons in the LoRA layers. The dropout method prevents overfitting by randomly ignoring (dropping out) neurons within a layer. A 10% dropout means that each neuron has a 10% chance of being dropped. A decimal number less than 1 for percentage, such as 0.1 for 10% 0.1
The following equation shows how the model calculates the totalTrainingSteps parameter.
totalTrainingSteps = (totalTrainingEpochs * size(trainingDataset)) / trainingBatchSize
In the preceding equation, the model ignores some rounding calculations.