Hugging Face's logo

Miayan
/
freelit-checkpoint

Diffusers
Safetensors
Model card Files
xet
 Community
Miayan commited on
Commit
c8fa304
·
verified ·
1 Parent(s): 682ca05

Add files using upload-large-folder tool

Browse files
Files changed (50) hide show
  1. albedo_estimator_best/diffusion_pytorch_model.safetensors +3 -0
  2. checkpoint_v1/checkpoint-235260/albedo_estimator/diffusion_pytorch_model.safetensors +3 -0
  3. checkpoint_v1/checkpoint-235260/controlnet/diffusion_pytorch_model.safetensors +3 -0
  4. checkpoint_v1/diffusion_pytorch_model.safetensors +3 -0
  5. checkpoint_v2/README.md +39 -0
  6. checkpoint_v2/checkpoint-235260/albedo_estimator/config.json +17 -0
  7. checkpoint_v2/checkpoint-235260/controlnet/config.json +56 -0
  8. checkpoint_v2/checkpoint-235260/custom_encoder/config.json +9 -0
  9. checkpoint_v2/checkpoint-235260/custom_encoder/diffusion_pytorch_model.safetensors +3 -0
  10. checkpoint_v2/checkpoint-235260/random_states_0.pkl +3 -0
  11. checkpoint_v2/checkpoint-235260/scheduler.bin +3 -0
  12. checkpoint_v2/config.json +56 -0
  13. checkpoint_v2/config.yaml +144 -0
  14. train_b1_lum/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors +3 -0
  15. train_b1_lum/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors +3 -0
  16. train_b1_lum/diffusion_pytorch_model.safetensors +3 -0
  17. train_b2_recon/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors +3 -0
  18. train_b2_recon/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors +3 -0
  19. train_b2_recon/diffusion_pytorch_model.safetensors +3 -0
  20. train_b3_cons/checkpoint-100000/albedo_estimator/config.json +17 -0
  21. train_b3_cons/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors +3 -0
  22. train_b3_cons/checkpoint-100000/controlnet/config.json +56 -0
  23. train_b3_cons/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors +3 -0
  24. train_b3_cons/checkpoint-100000/custom_encoder/config.json +9 -0
  25. train_b3_cons/checkpoint-100000/custom_encoder/diffusion_pytorch_model.safetensors +3 -0
  26. train_b3_cons/diffusion_pytorch_model.safetensors +3 -0
  27. train_b4_all/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors +3 -0
  28. train_b4_all/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors +3 -0
  29. train_b4_all/diffusion_pytorch_model.safetensors +3 -0
  30. train_r1_diff/README.md +39 -0
  31. train_r1_diff/checkpoint-100001/controlnet/config.json +57 -0
  32. train_r1_diff/checkpoint-100001/custom_encoder/config.json +10 -0
  33. train_r1_diff/checkpoint-100001/custom_encoder/diffusion_pytorch_model.safetensors +3 -0
  34. train_r1_diff/checkpoint-100001/random_states_0.pkl +3 -0
  35. train_r1_diff/checkpoint-100001/scheduler.bin +3 -0
  36. train_r1_diff/config.json +57 -0
  37. train_r1_diff/config.yaml +130 -0
  38. train_r1_diff/diffusion_pytorch_model.safetensors +3 -0
  39. train_r2_diff_ctrl/checkpoint-100001/controlnet/diffusion_pytorch_model.safetensors +3 -0
  40. train_r2_diff_ctrl/diffusion_pytorch_model.safetensors +3 -0
  41. train_r3_diff_ctrl_phys/README.md +39 -0
  42. train_r3_diff_ctrl_phys/checkpoint-100000/controlnet/config.json +56 -0
  43. train_r3_diff_ctrl_phys/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors +3 -0
  44. train_r3_diff_ctrl_phys/checkpoint-100000/custom_encoder/config.json +9 -0
  45. train_r3_diff_ctrl_phys/checkpoint-100000/custom_encoder/diffusion_pytorch_model.safetensors +3 -0
  46. train_r3_diff_ctrl_phys/checkpoint-100000/random_states_0.pkl +3 -0
  47. train_r3_diff_ctrl_phys/checkpoint-100000/scheduler.bin +3 -0
  48. train_r3_diff_ctrl_phys/config.json +56 -0
  49. train_r3_diff_ctrl_phys/config.yaml +130 -0
  50. train_r3_diff_ctrl_phys/diffusion_pytorch_model.safetensors +3 -0
albedo_estimator_best/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f47b24f36ad581dd4ea488c2f4de7aa39413265a3018bf283fbbbd1a9d477d3
3
+ size 422422660
checkpoint_v1/checkpoint-235260/albedo_estimator/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71031f7b051ffe470087dd14a896c25d75df3a5c0a32e74a8fdfb78527719028
3
+ size 422422660
checkpoint_v1/checkpoint-235260/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:324588a0bfdf61e038de35f20717920a99aed145a394f27b423dc8c36a1858f6
3
+ size 1461729992
checkpoint_v1/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:324588a0bfdf61e038de35f20717920a99aed145a394f27b423dc8c36a1858f6
3
+ size 1461729992
checkpoint_v2/README.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: stabilityai/stable-diffusion-2-1
3
+ library_name: diffusers
4
+ license: creativeml-openrail-m
5
+ inference: true
6
+ tags:
7
+ - stable-diffusion
8
+ - stable-diffusion-diffusers
9
+ - text-to-image
10
+ - diffusers
11
+ - controlnet
12
+ - diffusers-training
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the training script had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+
19
+ # controlnet-relighting_indoors
20
+
21
+ These are controlnet weights trained on stabilityai/stable-diffusion-2-1 with new type of conditioning.
22
+
23
+
24
+
25
+ ## Intended uses & limitations
26
+
27
+ #### How to use
28
+
29
+ ```python
30
+ # TODO: add an example code snippet for running this diffusion pipeline
31
+ ```
32
+
33
+ #### Limitations and bias
34
+
35
+ [TODO: provide examples of latent issues and potential remediations]
36
+
37
+ ## Training details
38
+
39
+ [TODO: describe the data used to train the model]
checkpoint_v2/checkpoint-235260/albedo_estimator/config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "AlbedoWrapper",
3
+ "_diffusers_version": "0.33.1",
4
+ "cfg": {
5
+ "enabled": true,
6
+ "frozen_components": [
7
+ "ord_model",
8
+ "iid_model",
9
+ "col_model"
10
+ ],
11
+ "load_stage": 3,
12
+ "trainable_components": [
13
+ "alb_model"
14
+ ],
15
+ "version": "v2"
16
+ }
17
+ }
checkpoint_v2/checkpoint-235260/controlnet/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": null,
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": null,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20,
12
+ 20
13
+ ],
14
+ "block_out_channels": [
15
+ 320,
16
+ 640,
17
+ 1280,
18
+ 1280
19
+ ],
20
+ "class_embed_type": null,
21
+ "conditioning_channels": 6,
22
+ "conditioning_embedding_out_channels": [
23
+ 16,
24
+ 32,
25
+ 96,
26
+ 256
27
+ ],
28
+ "controlnet_conditioning_channel_order": "rgb",
29
+ "cross_attention_dim": 1024,
30
+ "down_block_types": [
31
+ "CrossAttnDownBlock2D",
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "DownBlock2D"
35
+ ],
36
+ "downsample_padding": 1,
37
+ "encoder_hid_dim": null,
38
+ "encoder_hid_dim_type": null,
39
+ "flip_sin_to_cos": true,
40
+ "freq_shift": 0,
41
+ "global_pool_conditions": false,
42
+ "in_channels": 4,
43
+ "layers_per_block": 2,
44
+ "mid_block_scale_factor": 1,
45
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
46
+ "norm_eps": 1e-05,
47
+ "norm_num_groups": 32,
48
+ "num_attention_heads": null,
49
+ "num_class_embeds": null,
50
+ "only_cross_attention": false,
51
+ "projection_class_embeddings_input_dim": null,
52
+ "resnet_time_scale_shift": "default",
53
+ "transformer_layers_per_block": 1,
54
+ "upcast_attention": true,
55
+ "use_linear_projection": true
56
+ }
checkpoint_v2/checkpoint-235260/custom_encoder/config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Ba": 4,
3
+ "Bc": 4,
4
+ "Bi": 4,
5
+ "K": 4,
6
+ "_class_name": "CustomEncoder",
7
+ "_diffusers_version": "0.33.1",
8
+ "cross_dim": 1024
9
+ }
checkpoint_v2/checkpoint-235260/custom_encoder/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c85a0e54e1ea26e09b15b364e6a2e3fab9581bdc917c0a7ac20a76ae2961ab63
3
+ size 4541016
checkpoint_v2/checkpoint-235260/random_states_0.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c111a9e3aeed2f69c5f674696908bde40e2a8ed9a3ff423fd18357484b89c25
3
+ size 14757
checkpoint_v2/checkpoint-235260/scheduler.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f2bf6325ce12a3caa925c7153dfb341055dbe7f64dae17474e39c554ccc0f5f
3
+ size 1401
checkpoint_v2/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": null,
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": null,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20,
12
+ 20
13
+ ],
14
+ "block_out_channels": [
15
+ 320,
16
+ 640,
17
+ 1280,
18
+ 1280
19
+ ],
20
+ "class_embed_type": null,
21
+ "conditioning_channels": 6,
22
+ "conditioning_embedding_out_channels": [
23
+ 16,
24
+ 32,
25
+ 96,
26
+ 256
27
+ ],
28
+ "controlnet_conditioning_channel_order": "rgb",
29
+ "cross_attention_dim": 1024,
30
+ "down_block_types": [
31
+ "CrossAttnDownBlock2D",
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "DownBlock2D"
35
+ ],
36
+ "downsample_padding": 1,
37
+ "encoder_hid_dim": null,
38
+ "encoder_hid_dim_type": null,
39
+ "flip_sin_to_cos": true,
40
+ "freq_shift": 0,
41
+ "global_pool_conditions": false,
42
+ "in_channels": 4,
43
+ "layers_per_block": 2,
44
+ "mid_block_scale_factor": 1,
45
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
46
+ "norm_eps": 1e-05,
47
+ "norm_num_groups": 32,
48
+ "num_attention_heads": null,
49
+ "num_class_embeds": null,
50
+ "only_cross_attention": false,
51
+ "projection_class_embeddings_input_dim": null,
52
+ "resnet_time_scale_shift": "default",
53
+ "transformer_layers_per_block": 1,
54
+ "upcast_attention": true,
55
+ "use_linear_projection": true
56
+ }
checkpoint_v2/config.yaml ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ experiment:
2
+ project_name: relighting_indoors
3
+ output_dir: checkpoint_v2
4
+ logging_dir: logs
5
+ report_to: tensorboard
6
+ model:
7
+ pretrained_model_name_or_path: stable-diffusion-2-1
8
+ revision: null
9
+ variant: null
10
+ tokenizer_name: null
11
+ hf_repo_id: Miayan/project
12
+ hf_version: null
13
+ pretrain_unet_path: scribblelight_controlnet/checkpoint-10000
14
+ resume_from_checkpoint: latest
15
+ enable_ambient_cond: true
16
+ light_encoder:
17
+ cross_dim: 512
18
+ K: 4
19
+ Bi: 4
20
+ Bc: 4
21
+ Ba: 4
22
+ albedo_estimator:
23
+ enabled: true
24
+ version: v2
25
+ load_stage: 3
26
+ frozen_components:
27
+ - ord_model
28
+ - iid_model
29
+ - col_model
30
+ trainable_components:
31
+ - alb_model
32
+ data:
33
+ dataset_cache_dir: ./.hf_cache
34
+ data_hf_repo_id: Miayan/physical-relighting-dataset
35
+ data_split: test
36
+ resolution: 512
37
+ dataloader_num_workers: 0
38
+ condition_mode: v2
39
+ relighting_impl: ori
40
+ use_ambient_in_controlnet: false
41
+ use_color_on_lightmap: true
42
+ colors:
43
+ - - 255
44
+ - 255
45
+ - 255
46
+ - - 255
47
+ - 0
48
+ - 0
49
+ - - 0
50
+ - 255
51
+ - 0
52
+ - - 0
53
+ - 0
54
+ - 255
55
+ - - 255
56
+ - 255
57
+ - 0
58
+ - - 255
59
+ - 165
60
+ - 0
61
+ - - 128
62
+ - 0
63
+ - 128
64
+ - - 255
65
+ - 192
66
+ - 203
67
+ - - 0
68
+ - 255
69
+ - 255
70
+ - - 255
71
+ - 0
72
+ - 255
73
+ intensities:
74
+ - 0.0
75
+ - 0.1
76
+ - 0.2
77
+ - 0.4
78
+ - 0.7
79
+ - 1.0
80
+ training:
81
+ batch_size: 1
82
+ num_epochs: 20
83
+ max_train_steps: null
84
+ max_train_samples: null
85
+ seed: 42
86
+ mixed_precision: 'no'
87
+ allow_tf32: true
88
+ gradient_accumulation_steps: 1
89
+ checkpointing_steps: 5000
90
+ checkpoints_total_limit: 3
91
+ learning_rate: 5.0e-06
92
+ scale_lr: false
93
+ lr_scheduler: constant
94
+ lr_warmup_steps: 500
95
+ adam_beta1: 0.9
96
+ adam_beta2: 0.999
97
+ adam_weight_decay: 0.01
98
+ adam_epsilon: 1.0e-08
99
+ max_grad_norm: 1.0
100
+ use_8bit_adam: false
101
+ set_grads_to_none: false
102
+ losses:
103
+ phys_loss:
104
+ enabled: true
105
+ weight: 1.0
106
+ latent_loss:
107
+ enabled: true
108
+ weight: 1.0
109
+ image_loss:
110
+ enabled: true
111
+ weight: 1.0
112
+ ssim_loss:
113
+ enabled: false
114
+ weight: 0.2
115
+ area_loss:
116
+ enabled: false
117
+ weight: 10.0
118
+ intensity_loss:
119
+ enabled: false
120
+ weight: 1.0
121
+ recon_loss:
122
+ enabled: false
123
+ weight: 1.0
124
+ keep_aux_models_on_gpu: true
125
+ sup_structure_loss:
126
+ enabled: true
127
+ weight: 1.0
128
+ enable_decay: true
129
+ start_weight: 1.0
130
+ end_weight: 0.1
131
+ decay_steps: 5000
132
+ self_recon_loss:
133
+ enabled: true
134
+ mode: teacher
135
+ weight: 2.0
136
+ affine_alignment: true
137
+ epsilon: 0.05
138
+ staging:
139
+ switch_epoch: 5
140
+ stage1_weight: 2.0
141
+ stage2_weight: 0.5
142
+ consistency_loss:
143
+ enabled: true
144
+ weight: 1.0
train_b1_lum/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:78845419630b5f124156f6428636c25187a6039d9ab12a5b95dab825dd193dfa
3
+ size 422422660
train_b1_lum/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77efb9af63543194cb63c5839b3f1e3e8cd48157acd360b792491d5c4e9a26f5
3
+ size 1461729992
train_b1_lum/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77efb9af63543194cb63c5839b3f1e3e8cd48157acd360b792491d5c4e9a26f5
3
+ size 1461729992
train_b2_recon/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4965fc686e029c3ee587cb935b9d9d0c2def40e51b59a2757dcfa06fda83b74e
3
+ size 422422660
train_b2_recon/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b91c8e4cd32e37388579e4d3f199ff9944bfceaf321cab3e742c1dd7536bbded
3
+ size 1461729992
train_b2_recon/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b91c8e4cd32e37388579e4d3f199ff9944bfceaf321cab3e742c1dd7536bbded
3
+ size 1461729992
train_b3_cons/checkpoint-100000/albedo_estimator/config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "AlbedoWrapper",
3
+ "_diffusers_version": "0.33.1",
4
+ "cfg": {
5
+ "enabled": true,
6
+ "frozen_components": [
7
+ "ord_model",
8
+ "iid_model",
9
+ "col_model"
10
+ ],
11
+ "load_stage": 3,
12
+ "trainable_components": [
13
+ "alb_model"
14
+ ],
15
+ "version": "v2"
16
+ }
17
+ }
train_b3_cons/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:91592f4dc64b0cca93efda77d604e540aa86f6c2744b5abfcc076cd2ba1383a9
3
+ size 422422660
train_b3_cons/checkpoint-100000/controlnet/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": null,
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": null,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20,
12
+ 20
13
+ ],
14
+ "block_out_channels": [
15
+ 320,
16
+ 640,
17
+ 1280,
18
+ 1280
19
+ ],
20
+ "class_embed_type": null,
21
+ "conditioning_channels": 6,
22
+ "conditioning_embedding_out_channels": [
23
+ 16,
24
+ 32,
25
+ 96,
26
+ 256
27
+ ],
28
+ "controlnet_conditioning_channel_order": "rgb",
29
+ "cross_attention_dim": 1024,
30
+ "down_block_types": [
31
+ "CrossAttnDownBlock2D",
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "DownBlock2D"
35
+ ],
36
+ "downsample_padding": 1,
37
+ "encoder_hid_dim": null,
38
+ "encoder_hid_dim_type": null,
39
+ "flip_sin_to_cos": true,
40
+ "freq_shift": 0,
41
+ "global_pool_conditions": false,
42
+ "in_channels": 4,
43
+ "layers_per_block": 2,
44
+ "mid_block_scale_factor": 1,
45
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
46
+ "norm_eps": 1e-05,
47
+ "norm_num_groups": 32,
48
+ "num_attention_heads": null,
49
+ "num_class_embeds": null,
50
+ "only_cross_attention": false,
51
+ "projection_class_embeddings_input_dim": null,
52
+ "resnet_time_scale_shift": "default",
53
+ "transformer_layers_per_block": 1,
54
+ "upcast_attention": true,
55
+ "use_linear_projection": true
56
+ }
train_b3_cons/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:404182ffb6c32a9775f1da8d114fd9076a1d10104227bc245dc6cdd80bc52863
3
+ size 1461729992
train_b3_cons/checkpoint-100000/custom_encoder/config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Ba": 4,
3
+ "Bc": 4,
4
+ "Bi": 4,
5
+ "K": 4,
6
+ "_class_name": "CustomEncoder",
7
+ "_diffusers_version": "0.33.1",
8
+ "cross_dim": 1024
9
+ }
train_b3_cons/checkpoint-100000/custom_encoder/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce0eb3e75e59942f6fb030e5c72164383b8c326f3e48ae75128b890e74aae610
3
+ size 4541016
train_b3_cons/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:404182ffb6c32a9775f1da8d114fd9076a1d10104227bc245dc6cdd80bc52863
3
+ size 1461729992
train_b4_all/checkpoint-100000/albedo_estimator/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b5483cb15fb93b728a9cbff76156a693dbd83225d452d65ce5beed6943d4ecbc
3
+ size 422422660
train_b4_all/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13fc2f7440dea8e0f839127a5156dc6fd3b751865d1fdeb9502917c180095bc5
3
+ size 1461729992
train_b4_all/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13fc2f7440dea8e0f839127a5156dc6fd3b751865d1fdeb9502917c180095bc5
3
+ size 1461729992
train_r1_diff/README.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: stabilityai/stable-diffusion-2-1
3
+ library_name: diffusers
4
+ license: creativeml-openrail-m
5
+ inference: true
6
+ tags:
7
+ - stable-diffusion
8
+ - stable-diffusion-diffusers
9
+ - text-to-image
10
+ - diffusers
11
+ - controlnet
12
+ - diffusers-training
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the training script had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+
19
+ # controlnet-relighting_indoors
20
+
21
+ These are controlnet weights trained on stabilityai/stable-diffusion-2-1 with new type of conditioning.
22
+
23
+
24
+
25
+ ## Intended uses & limitations
26
+
27
+ #### How to use
28
+
29
+ ```python
30
+ # TODO: add an example code snippet for running this diffusion pipeline
31
+ ```
32
+
33
+ #### Limitations and bias
34
+
35
+ [TODO: provide examples of latent issues and potential remediations]
36
+
37
+ ## Training details
38
+
39
+ [TODO: describe the data used to train the model]
train_r1_diff/checkpoint-100001/controlnet/config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "_name_or_path": "train_r1_diff/checkpoint-10000",
5
+ "act_fn": "silu",
6
+ "addition_embed_type": null,
7
+ "addition_embed_type_num_heads": 64,
8
+ "addition_time_embed_dim": null,
9
+ "attention_head_dim": [
10
+ 5,
11
+ 10,
12
+ 20,
13
+ 20
14
+ ],
15
+ "block_out_channels": [
16
+ 320,
17
+ 640,
18
+ 1280,
19
+ 1280
20
+ ],
21
+ "class_embed_type": null,
22
+ "conditioning_channels": 6,
23
+ "conditioning_embedding_out_channels": [
24
+ 16,
25
+ 32,
26
+ 96,
27
+ 256
28
+ ],
29
+ "controlnet_conditioning_channel_order": "rgb",
30
+ "cross_attention_dim": 1024,
31
+ "down_block_types": [
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "CrossAttnDownBlock2D",
35
+ "DownBlock2D"
36
+ ],
37
+ "downsample_padding": 1,
38
+ "encoder_hid_dim": null,
39
+ "encoder_hid_dim_type": null,
40
+ "flip_sin_to_cos": true,
41
+ "freq_shift": 0,
42
+ "global_pool_conditions": false,
43
+ "in_channels": 4,
44
+ "layers_per_block": 2,
45
+ "mid_block_scale_factor": 1,
46
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
47
+ "norm_eps": 1e-05,
48
+ "norm_num_groups": 32,
49
+ "num_attention_heads": null,
50
+ "num_class_embeds": null,
51
+ "only_cross_attention": false,
52
+ "projection_class_embeddings_input_dim": null,
53
+ "resnet_time_scale_shift": "default",
54
+ "transformer_layers_per_block": 1,
55
+ "upcast_attention": true,
56
+ "use_linear_projection": true
57
+ }
train_r1_diff/checkpoint-100001/custom_encoder/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Ba": 4,
3
+ "Bc": 4,
4
+ "Bi": 4,
5
+ "K": 4,
6
+ "_class_name": "CustomEncoder",
7
+ "_diffusers_version": "0.33.1",
8
+ "_name_or_path": "train_r1_diff/checkpoint-10000",
9
+ "cross_dim": 1024
10
+ }
train_r1_diff/checkpoint-100001/custom_encoder/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e68c3a7f7cb7bf47dbf87139dd42b94f576b3e58d3f047d387bd31a6f9da1763
3
+ size 4541016
train_r1_diff/checkpoint-100001/random_states_0.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de47f527a99690bd0feb56a71f0b6c18439516a89c011c8ddddb45550724d3d4
3
+ size 14821
train_r1_diff/checkpoint-100001/scheduler.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4c83d1406871888f856672fc77fa480ebd954fd28f0b50673daff4f4da91737
3
+ size 1401
train_r1_diff/config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "_name_or_path": "train_r1_diff/checkpoint-10000",
5
+ "act_fn": "silu",
6
+ "addition_embed_type": null,
7
+ "addition_embed_type_num_heads": 64,
8
+ "addition_time_embed_dim": null,
9
+ "attention_head_dim": [
10
+ 5,
11
+ 10,
12
+ 20,
13
+ 20
14
+ ],
15
+ "block_out_channels": [
16
+ 320,
17
+ 640,
18
+ 1280,
19
+ 1280
20
+ ],
21
+ "class_embed_type": null,
22
+ "conditioning_channels": 6,
23
+ "conditioning_embedding_out_channels": [
24
+ 16,
25
+ 32,
26
+ 96,
27
+ 256
28
+ ],
29
+ "controlnet_conditioning_channel_order": "rgb",
30
+ "cross_attention_dim": 1024,
31
+ "down_block_types": [
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "CrossAttnDownBlock2D",
35
+ "DownBlock2D"
36
+ ],
37
+ "downsample_padding": 1,
38
+ "encoder_hid_dim": null,
39
+ "encoder_hid_dim_type": null,
40
+ "flip_sin_to_cos": true,
41
+ "freq_shift": 0,
42
+ "global_pool_conditions": false,
43
+ "in_channels": 4,
44
+ "layers_per_block": 2,
45
+ "mid_block_scale_factor": 1,
46
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
47
+ "norm_eps": 1e-05,
48
+ "norm_num_groups": 32,
49
+ "num_attention_heads": null,
50
+ "num_class_embeds": null,
51
+ "only_cross_attention": false,
52
+ "projection_class_embeddings_input_dim": null,
53
+ "resnet_time_scale_shift": "default",
54
+ "transformer_layers_per_block": 1,
55
+ "upcast_attention": true,
56
+ "use_linear_projection": true
57
+ }
train_r1_diff/config.yaml ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ experiment:
2
+ project_name: relighting_indoors
3
+ output_dir: train_r1_diff
4
+ logging_dir: logs
5
+ report_to: tensorboard
6
+ model:
7
+ pretrained_model_name_or_path: stable-diffusion-2-1
8
+ revision: null
9
+ variant: null
10
+ tokenizer_name: null
11
+ hf_repo_id: Miayan/project
12
+ hf_version: null
13
+ pretrain_unet_path: scribblelight_controlnet/checkpoint-10000
14
+ resume_from_checkpoint: latest
15
+ enable_ambient_cond: true
16
+ light_encoder:
17
+ cross_dim: 768
18
+ K: 4
19
+ Bi: 4
20
+ Bc: 4
21
+ Ba: 4
22
+ albedo_estimator:
23
+ enabled: false
24
+ version: v2
25
+ load_stage: 3
26
+ frozen_components:
27
+ - ord_model
28
+ - iid_model
29
+ - col_model
30
+ trainable_components: []
31
+ data:
32
+ dataset_cache_dir: ./.hf_cache
33
+ data_hf_repo_id: Miayan/physical-relighting-dataset
34
+ data_split: test
35
+ resolution: 512
36
+ dataloader_num_workers: 0
37
+ condition_mode: v1
38
+ relighting_impl: ori
39
+ use_ambient_in_controlnet: false
40
+ use_color_on_lightmap: true
41
+ colors:
42
+ - - 255
43
+ - 255
44
+ - 255
45
+ - - 255
46
+ - 0
47
+ - 0
48
+ - - 0
49
+ - 255
50
+ - 0
51
+ - - 0
52
+ - 0
53
+ - 255
54
+ - - 255
55
+ - 255
56
+ - 0
57
+ - - 255
58
+ - 165
59
+ - 0
60
+ - - 128
61
+ - 0
62
+ - 128
63
+ - - 255
64
+ - 192
65
+ - 203
66
+ - - 0
67
+ - 255
68
+ - 255
69
+ - - 255
70
+ - 0
71
+ - 255
72
+ intensities:
73
+ - 0.0
74
+ - 0.2
75
+ - 0.4
76
+ - 0.6
77
+ - 0.8
78
+ - 1.0
79
+ training:
80
+ batch_size: 1
81
+ num_epochs: 20
82
+ max_train_steps: 100000
83
+ max_train_samples: null
84
+ seed: 42
85
+ mixed_precision: 'no'
86
+ allow_tf32: true
87
+ gradient_accumulation_steps: 1
88
+ checkpointing_steps: 5000
89
+ checkpoints_total_limit: 4
90
+ learning_rate: 5.0e-06
91
+ scale_lr: false
92
+ lr_scheduler: constant
93
+ lr_warmup_steps: 500
94
+ adam_beta1: 0.9
95
+ adam_beta2: 0.999
96
+ adam_weight_decay: 0.01
97
+ adam_epsilon: 1.0e-08
98
+ max_grad_norm: 1.0
99
+ use_8bit_adam: false
100
+ set_grads_to_none: false
101
+ losses:
102
+ phys_loss:
103
+ enabled: false
104
+ weight: 1.0
105
+ latent_loss:
106
+ enabled: true
107
+ weight: 1.0
108
+ image_loss:
109
+ enabled: false
110
+ weight: 1.0
111
+ sup_structure_loss:
112
+ enabled: false
113
+ weight: 1.0
114
+ enable_decay: true
115
+ start_weight: 1.0
116
+ end_weight: 0.1
117
+ decay_steps: 5000
118
+ self_recon_loss:
119
+ enabled: false
120
+ mode: teacher
121
+ weight: 2.0
122
+ affine_alignment: true
123
+ epsilon: 0.05
124
+ staging:
125
+ switch_epoch: 5
126
+ stage1_weight: 2.0
127
+ stage2_weight: 0.5
128
+ consistency_loss:
129
+ enabled: false
130
+ weight: 1.0
train_r1_diff/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2be0512584969569c0714019fbf4018506c09a9f94cb518f895e0f63078e0bd6
3
+ size 1461729992
train_r2_diff_ctrl/checkpoint-100001/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:439020241fddfc05005e9be7d0feb74bf5e6af92e1852e740662cdaecd991d05
3
+ size 1461729992
train_r2_diff_ctrl/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:439020241fddfc05005e9be7d0feb74bf5e6af92e1852e740662cdaecd991d05
3
+ size 1461729992
train_r3_diff_ctrl_phys/README.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: stabilityai/stable-diffusion-2-1
3
+ library_name: diffusers
4
+ license: creativeml-openrail-m
5
+ inference: true
6
+ tags:
7
+ - stable-diffusion
8
+ - stable-diffusion-diffusers
9
+ - text-to-image
10
+ - diffusers
11
+ - controlnet
12
+ - diffusers-training
13
+ ---
14
+
15
+ <!-- This model card has been generated automatically according to the information the training script had access to. You
16
+ should probably proofread and complete it, then remove this comment. -->
17
+
18
+
19
+ # controlnet-relighting_indoors
20
+
21
+ These are controlnet weights trained on stabilityai/stable-diffusion-2-1 with new type of conditioning.
22
+
23
+
24
+
25
+ ## Intended uses & limitations
26
+
27
+ #### How to use
28
+
29
+ ```python
30
+ # TODO: add an example code snippet for running this diffusion pipeline
31
+ ```
32
+
33
+ #### Limitations and bias
34
+
35
+ [TODO: provide examples of latent issues and potential remediations]
36
+
37
+ ## Training details
38
+
39
+ [TODO: describe the data used to train the model]
train_r3_diff_ctrl_phys/checkpoint-100000/controlnet/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": null,
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": null,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20,
12
+ 20
13
+ ],
14
+ "block_out_channels": [
15
+ 320,
16
+ 640,
17
+ 1280,
18
+ 1280
19
+ ],
20
+ "class_embed_type": null,
21
+ "conditioning_channels": 6,
22
+ "conditioning_embedding_out_channels": [
23
+ 16,
24
+ 32,
25
+ 96,
26
+ 256
27
+ ],
28
+ "controlnet_conditioning_channel_order": "rgb",
29
+ "cross_attention_dim": 1024,
30
+ "down_block_types": [
31
+ "CrossAttnDownBlock2D",
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "DownBlock2D"
35
+ ],
36
+ "downsample_padding": 1,
37
+ "encoder_hid_dim": null,
38
+ "encoder_hid_dim_type": null,
39
+ "flip_sin_to_cos": true,
40
+ "freq_shift": 0,
41
+ "global_pool_conditions": false,
42
+ "in_channels": 4,
43
+ "layers_per_block": 2,
44
+ "mid_block_scale_factor": 1,
45
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
46
+ "norm_eps": 1e-05,
47
+ "norm_num_groups": 32,
48
+ "num_attention_heads": null,
49
+ "num_class_embeds": null,
50
+ "only_cross_attention": false,
51
+ "projection_class_embeddings_input_dim": null,
52
+ "resnet_time_scale_shift": "default",
53
+ "transformer_layers_per_block": 1,
54
+ "upcast_attention": true,
55
+ "use_linear_projection": true
56
+ }
train_r3_diff_ctrl_phys/checkpoint-100000/controlnet/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be86b95516d1d2fe585ecee2e63f8bc93e0c991559731a71ae42679cbba72e59
3
+ size 1461729992
train_r3_diff_ctrl_phys/checkpoint-100000/custom_encoder/config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "Ba": 4,
3
+ "Bc": 4,
4
+ "Bi": 4,
5
+ "K": 4,
6
+ "_class_name": "CustomEncoder",
7
+ "_diffusers_version": "0.33.1",
8
+ "cross_dim": 1024
9
+ }
train_r3_diff_ctrl_phys/checkpoint-100000/custom_encoder/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1312df18dfda4efbef4940c33a41f070b6c575d8db6d17bdae04e38032082b42
3
+ size 4541016
train_r3_diff_ctrl_phys/checkpoint-100000/random_states_0.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3629b14093a195ad50669e7aa045d6d7fd34c6a94e7ea5d2158a347884975bfd
3
+ size 14821
train_r3_diff_ctrl_phys/checkpoint-100000/scheduler.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0a6567d2bf48dae2a2de818bd363ae868ced24a1ad4a2cb7551d2ce4c1edbac
3
+ size 1401
train_r3_diff_ctrl_phys/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.33.1",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": null,
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": null,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20,
12
+ 20
13
+ ],
14
+ "block_out_channels": [
15
+ 320,
16
+ 640,
17
+ 1280,
18
+ 1280
19
+ ],
20
+ "class_embed_type": null,
21
+ "conditioning_channels": 6,
22
+ "conditioning_embedding_out_channels": [
23
+ 16,
24
+ 32,
25
+ 96,
26
+ 256
27
+ ],
28
+ "controlnet_conditioning_channel_order": "rgb",
29
+ "cross_attention_dim": 1024,
30
+ "down_block_types": [
31
+ "CrossAttnDownBlock2D",
32
+ "CrossAttnDownBlock2D",
33
+ "CrossAttnDownBlock2D",
34
+ "DownBlock2D"
35
+ ],
36
+ "downsample_padding": 1,
37
+ "encoder_hid_dim": null,
38
+ "encoder_hid_dim_type": null,
39
+ "flip_sin_to_cos": true,
40
+ "freq_shift": 0,
41
+ "global_pool_conditions": false,
42
+ "in_channels": 4,
43
+ "layers_per_block": 2,
44
+ "mid_block_scale_factor": 1,
45
+ "mid_block_type": "UNetMidBlock2DCrossAttn",
46
+ "norm_eps": 1e-05,
47
+ "norm_num_groups": 32,
48
+ "num_attention_heads": null,
49
+ "num_class_embeds": null,
50
+ "only_cross_attention": false,
51
+ "projection_class_embeddings_input_dim": null,
52
+ "resnet_time_scale_shift": "default",
53
+ "transformer_layers_per_block": 1,
54
+ "upcast_attention": true,
55
+ "use_linear_projection": true
56
+ }
train_r3_diff_ctrl_phys/config.yaml ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ experiment:
2
+ project_name: relighting_indoors
3
+ output_dir: train_r3_diff_ctrl_phys
4
+ logging_dir: logs
5
+ report_to: tensorboard
6
+ model:
7
+ pretrained_model_name_or_path: stable-diffusion-2-1
8
+ revision: null
9
+ variant: null
10
+ tokenizer_name: null
11
+ hf_repo_id: Miayan/project
12
+ hf_version: null
13
+ pretrain_unet_path: scribblelight_controlnet/checkpoint-10000
14
+ resume_from_checkpoint: latest
15
+ enable_ambient_cond: true
16
+ light_encoder:
17
+ cross_dim: 768
18
+ K: 4
19
+ Bi: 4
20
+ Bc: 4
21
+ Ba: 4
22
+ albedo_estimator:
23
+ enabled: false
24
+ version: v2
25
+ load_stage: 3
26
+ frozen_components:
27
+ - ord_model
28
+ - iid_model
29
+ - col_model
30
+ trainable_components: []
31
+ data:
32
+ dataset_cache_dir: ./.hf_cache
33
+ data_hf_repo_id: Miayan/physical-relighting-dataset
34
+ data_split: test
35
+ resolution: 512
36
+ dataloader_num_workers: 0
37
+ condition_mode: v1
38
+ relighting_impl: ori
39
+ use_ambient_in_controlnet: false
40
+ use_color_on_lightmap: true
41
+ colors:
42
+ - - 255
43
+ - 255
44
+ - 255
45
+ - - 255
46
+ - 0
47
+ - 0
48
+ - - 0
49
+ - 255
50
+ - 0
51
+ - - 0
52
+ - 0
53
+ - 255
54
+ - - 255
55
+ - 255
56
+ - 0
57
+ - - 255
58
+ - 165
59
+ - 0
60
+ - - 128
61
+ - 0
62
+ - 128
63
+ - - 255
64
+ - 192
65
+ - 203
66
+ - - 0
67
+ - 255
68
+ - 255
69
+ - - 255
70
+ - 0
71
+ - 255
72
+ intensities:
73
+ - 0.0
74
+ - 0.2
75
+ - 0.4
76
+ - 0.6
77
+ - 0.8
78
+ - 1.0
79
+ training:
80
+ batch_size: 1
81
+ num_epochs: 20
82
+ max_train_steps: 100000
83
+ max_train_samples: null
84
+ seed: 42
85
+ mixed_precision: 'no'
86
+ allow_tf32: true
87
+ gradient_accumulation_steps: 1
88
+ checkpointing_steps: 5000
89
+ checkpoints_total_limit: 4
90
+ learning_rate: 5.0e-06
91
+ scale_lr: false
92
+ lr_scheduler: constant
93
+ lr_warmup_steps: 500
94
+ adam_beta1: 0.9
95
+ adam_beta2: 0.999
96
+ adam_weight_decay: 0.01
97
+ adam_epsilon: 1.0e-08
98
+ max_grad_norm: 1.0
99
+ use_8bit_adam: false
100
+ set_grads_to_none: false
101
+ losses:
102
+ phys_loss:
103
+ enabled: true
104
+ weight: 1.0
105
+ latent_loss:
106
+ enabled: true
107
+ weight: 1.0
108
+ image_loss:
109
+ enabled: true
110
+ weight: 1.0
111
+ sup_structure_loss:
112
+ enabled: false
113
+ weight: 1.0
114
+ enable_decay: true
115
+ start_weight: 1.0
116
+ end_weight: 0.1
117
+ decay_steps: 5000
118
+ self_recon_loss:
119
+ enabled: false
120
+ mode: teacher
121
+ weight: 2.0
122
+ affine_alignment: true
123
+ epsilon: 0.05
124
+ staging:
125
+ switch_epoch: 5
126
+ stage1_weight: 2.0
127
+ stage2_weight: 0.5
128
+ consistency_loss:
129
+ enabled: false
130
+ weight: 1.0
train_r3_diff_ctrl_phys/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be86b95516d1d2fe585ecee2e63f8bc93e0c991559731a71ae42679cbba72e59
3
+ size 1461729992