Question: dart_plugin_registrant.dart + AOT + --obfuscation?

@jwinarske Thanks for filing the bug. For the record, @stuartmorgan is the TL for this area of the code so he's the one who should make the final call on this.

It sounds like there's an ambiguity in your request. We support several Linux-based platforms, for example Android and GTK. Plugins for different platforms are quite different (for example, plugins on Android are typically in Java, whereas on GTK they are in C!).

The core problem, I think, is that we don't yet have extensible logic in the tool so that you can easily add new platforms. We should definitely work on that. URL is where we're tracking that issue. You can discuss this in #hackers-tool on our Discord. As a heavy user of a custom embedding, you may be interested in getting involved in this work; we haven't really started on it in earnest yet but it is an area we are very interested in.

dart_plugin_registrant.dart + AOT + --obfuscation
### Is there an existing issue for this?

- [X] I have searched the existing issues
- [X] I have read the guide to filing a bug

### Steps to reproduce

Manually build AOT components where you can control use of the --obfuscate flag.

Build sequences:
 URL
 URL

### Expected results

correctly register plugins regardless of obfuscate flag

### Actual results

without passing --obfuscate to `gen_snapshot`:
* Flutter Package examples - correctly register plugins
* Gallery - does not register plugins
* Wonderous - does not register plugins

passing --obfuscate to `gen_snapshot`:
* Flutter Package examples - correctly register plugins
* Gallery - correctly register plugins
* Wonderous - correctly register plugins

Dumping symbols from AOT in both cases using readelf, the expected entry points are present:

working

 11673: 000000000059914c 116 FUNC LOCAL DEFAULT 7 _PluginRegistrant.register
 11674: 00000000005991c0 44 FUNC LOCAL DEFAULT 7 _PluginRegistrant.register
 11675: 00000000005991ec 8 FUNC LOCAL DEFAULT 7 dartPluginRegistrantLibrary
 11676: 00000000005991f4 8 FUNC LOCAL DEFAULT 7 dartPluginRegistrantLibrary
 11677: 00000000005991fc 48 FUNC LOCAL DEFAULT 7 main

not working

131825: 0000000001e991e0 8 FUNC LOCAL DEFAULT 7 dartPluginRegistrantLibrary
131826: 0000000001e991e8 8 FUNC LOCAL DEFAULT 7 dartPluginRegistrantLibrary
131827: 0000000001e991f0 320 FUNC LOCAL DEFAULT 7 _PluginRegistrant.register
131828: 0000000001e99330 44 FUNC LOCAL DEFAULT 7 _PluginRegistrant.register
131829: 0000000001e9935c 96 FUNC LOCAL DEFAULT 7 main
131830: 0000000001e993bc 44 FUNC LOCAL DEFAULT 7 main

### Code sample

NA

### Screenshots or Video

NA

### Logs

NA

### Flutter Doctor output

 Doctor output

console
[!] Flutter (Channel [user-branch], 3.13.9, on Fedora Linux 38 (Workstation Edition) 6.5.8-200.fc38.x86_64, locale en_US.UTF-8)
 ! Flutter version 3.13.9 on channel [user-branch] at /mnt/raid10/workspace-automation/flutter
 Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
 If that doesn't fix the issue, reinstall Flutter by following instructions at URL
 ! Upstream repository unknown source is not a standard remote.
 Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
 • Framework revision d211f42860 (12 days ago), 2023-10-25 13:42:25 -0700
 • Engine revision 0545f8705d
 • Dart version 3.1.5
 • DevTools version 2.25.0
 • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Linux toolchain - develop for Linux desktop
 • clang version 16.0.6 (Fedora 16.0.6-3.fc38)
 • cmake version 3.27.7
 • ninja version 1.11.1
 • pkg-config version 1.8.0

[✓] VS Code (version 1.84.1)
 • VS Code at /usr/share/code
 • Flutter extension version 3.76.0

[✓] Connected device (2 available)
 • Linux (desktop) • linux • linux-x64 • Fedora Linux 38 (Workstation Edition) 6.5.8-200.fc38.x86_64
 • Toyota homescreen (mobile) • desktop-homescreen • linux-x64 • homescreen x86_64

[✓] Network resources
 • All expected network resources are available.

! Doctor found issues in 1 category.
List of custom devices in "/mnt/raid10/workspace-automation/.config/flutter/custom_devices.json":
 id: desktop-homescreen, label: Toyota homescreen, enabled: true
[joel@air workspace-automation]$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel [user-branch], 3.13.9, on Fedora Linux 38 (Workstation Edition) 6.5.8-200.fc38.x86_64, locale en_US.UTF-8)
 ! Flutter version 3.13.9 on channel [user-branch] at /mnt/raid10/workspace-automation/flutter
 Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
 If that doesn't fix the issue, reinstall Flutter by following instructions at URL
 ! Upstream repository unknown source is not a standard remote.
 Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
[✓] Linux toolchain - develop for Linux desktop
[✓] VS Code (version 1.84.1)
[✓] Connected device (2 available)
[✓] Network resources

! Doctor found issues in 1 category.

Just to confirm: are you saying that if you pass `--obfuscate` to `gen_snapshot` then thing work _correctly_ and if you do **not** pass `--obfuscate` then things work _incorrectly_? I am a bit puzzled.

@mraleph

Correct. It's as if in the non-working case it's taking an alternate entry. Non-working cases default to platform channel.

The other pattern I determined is the working cases are all "local" packages. The non-working case use "remote" packages.

Not sure if Android and iOS default to having obfuscation set. If so that might explain why it's now showing up.

There was another issue a while back about obfuscation breaking dart registration I was looking at. Which introduced the --source solution.

@jwinarske , would it be possible for you to make actual repro out of URL for example? This should help understand better where in the stack there is obfuscate vs non-obfuscate confusion.

cc'ing @stuartmorgan since it seems to be in plugins/packages realm.

It is unclear to me what the expected outcome of this issue is given that URL and URL both explained that what this issue lists under "expected results" is in fact a bug that we intend to fix.

As far as I can tell this is a duplicate of URL

@stuartmorgan this is a new issue. Which in everything I've read should not behave this way. There is also another issue that drove the --source option; where dart plugin registrant was ignored only when obfuscated was set. Which indicated this might be a leftover corner case. Cross compiling an AOT by script or manually on a host machine is a valid use case. Dart pluginn registrant (vm entry) works today if I enable obsfucation when dealing with "remote package" definitions. In all cases of "local packages" plug-in references work with or without obfuscation.

It would be helpful if you could add more detail to the STR then. Right now the STR section is just links to a script that appear to be part of a build process rather than actual steps, so I was trying to extrapolate from that, and it appeared that you were describing a situation where—when building for a third-party embedding—the `flutter`-tool-generated first-party registrant code is sometimes being run and sometimes not (which in the important respects would be the same as URL

I understand that the exact details of what you are saying is different between the case where the registration runs and the case where it doesn't are different. That's why I said it would be the same "in the important respects".

I'm still not clear whether my understanding of the issue described in my last comment is correct or not. Please provide actual *steps* to reproduce the issue so that it's clear what the scenario is here.

Linux AOT via `flutter build bundle`
Building an application on a Linux machine using `flutter build bundle` is problematic.

1) The debug build will register and run the Linux Dart code. Stuart Morgan has stated this is definitely a bug. I'm building the bundle on Linux why would it not include Linux platform specific code?
2) Using the built bundle to create an AOT will drop the Linux Dart code, and the Linux specific Dart code is not executed.

When running `flutter build bundle` on a Linux machine (platform type is literally Linux), why would it not include the Linux specific Dart code?

Given `flutter build linux` is currently not capable of cross compiling, one cannot run `flutter build linux` and use the libapp.so (it's host only).

This is impacting developers targeting AGL Flutter. We (AGL/Toyota) want support for Linux specific Dart when using `flutter build bundle` for both Debug images and AOT images.

We want `flutter build bundle` and AOT generation to build/run the Linux specific Dart code regardless of the native CMake stuff.

@Hixie @timsneath @cmc5788

I have answered this at length in both URL and our discussion in Discord. Capturing the core bit from the latter:

Continuing to conflate "Linux-specific Dart" and "Flutter-1P-Linux-embedding-specific Dart" isn't constructive.

That support already exists; you can write Linux-specific Dart using `Platform.isLinux` checks.

If what you want is the ability to auto-generate Dart plugin registration code for your custom embedding, as I've said before we're happy to review designs for such a system, as part of the custom embedding hook support in the tool.

What you are specifically requesting here though, which is for your embedding to get Flutter-1P-Linux-embedding-specific Dart, is a wontfix.

@Hixie Thanks for the constructive response, I'll look into the referenced issue and Discord channel.

Just a heads up. This was a tooling issue. Resolved in my AOT generation sequence:
 URL

What I did uncover in my investigation to resolving this was that some apps worked fine with gen_snapshot obfuscation disabled, some did not. The Flutter Package examples worked, while Gallery and Wonders did not. Enabling obfuscation enabled Gallery and Wonders to work as expected. I raised this on the hacker-tools discord channel.


Question: General questions?

@TPCollings, I was wondering if you had the chance to test the updates? If not, I can merge them to the `main` branch and you can just test it there whenever you get the chance to.

General questions
Hi Soroosh,

Sorry I've got some more general questions about OCSmesh and didn't know the best way to contact you. I hope this is okay.

Is there any functionality in OCSmesh to allow a review of the CFL number, to ensure the grid won't create numerical diffusion?

Cheers,

Tom

Hi Tom, right now such a capability does **not** exist. However, I'd like to add new functionalities (where possible) based on the need.

OCSMesh uses `Jigasw`'s Python wrapper as the meshing engine, and as far as I know `Jigsaw` doesn't accept custom error function. So the best way I can think of to add this functionality is to fiddle with the size function. A method can be added to the `hfun` objects to make sure the specified size is within a given bound or criteria. Right now size constraint based on the topobathy exists; it shouldn't be too hard to add one that gets a reference velocity and timestep to constraint based on CFL as well. For example depending on the model being implicit or explicit you'd do something like:

python
hfun = ...
hfun.add_contours(...)
...
hfun.add_cfl_constraint(cfl_upper=1 , cfl_lower=-np.inf, ref_vel=10, ref_dt=150, rate=0.1)
...

which means *after* applying all refinements, it will check the specified sizes and if it falls outside the specified CFL bounds it will modify it and for regions adjacent to the adjusted location it will propagate based on the distance from the adjusted location.

Do you have a specific way of achieving the CFL check in mind? Is what I described above useful for your use case?

Hi Soroosh, thanks for your quick response.

That's a good suggestion you've made. I wonder how dependent it would be on the reference velocity and dt values that are chosen.

Having read the documentation of OceanMesh2d, they deal with the CFL number not by changing the grid size, but by suggesting the timestep for the model run using the minimum current number. This could potentially be an easier solution to changing the mesh itself?

They also also estimate u as a function of sea surface elevation and depth (|u| = ssh/sqrt(g/h)), taking ssh as 2m which is probably suitable for most regions. I've included an image of the documentation as they describe it better than I can .

Having some form of CFL check would be extremely useful. I'm looking to generate meshes over large basins, so manually estimating the CFL number for selected points isn't really feasible.

image

@TPCollings, I think it really depends on what solver you're using and how you want to set it up; for example for implicit solvers we might actually have to be careful for Courant number to remain above 1. In either case, one either has to choose what their Courant number is and what time step they'd like to use, then find the mesh size from it; or already have the mesh size and Courant number and would like to know what time step is suitable.

- For the first functionality it makes sense to have a method that manipulates the *size function* - as described in this comment - where it gets input η or velocity reference as well as dt and the bound on Courant number.
- For the second functionality it is more sensible to add a method to *mesh* object. So that when you already have a mesh, pass in the bounds on Courant number and get an estimate for dt.

@WPringle I haven't really worked with OceanMesh2D, can you please comment on this? Does what I'm saying make sense based on your experience?

@SorooshMani-NOAA Yeah I see your point. I'll be using Schism (more specifically PySchism), so I believe the minimum Courant number shouldn't be less than 0.4 for most modelling applications. How long do you think it might take to add the first functionality described above, which involves manipulating the size function?

Yes there are two ways to do it, both we try to do in OM2D. First is try to manipulate the size function beforehand estimating what the Courant number would be based on size function and depth there and what timestep you want to use. Second way is to post-process the mesh based on final Courant number by deleting or adding in new points and locally remeshing. We did add the second for SCHISM type meshes where we want to provide a lower bound on Courant number, and so therefore we try to add in more points, as well as deleting points to make the upper bound on Courant number which is usually more important for ADCIRC.

@SorooshMani-NOAA That would be great thanks. There's no super urgency, but it would be great to have it before the end of the month. Cheers!

@WPringle, I have a technical question. Why is the Courant number calculation in the referenced text from OceanMesh2D calculated with an additional $\sqrt{g H}$? Is this some kind of *corrected* Courant number for shallow water equation?

Yes. Courant number can be calculated from the characteristic velocity: |u| + sqrt(gh). Here, u is the particle velocity which is unknown but from linear wave theory can be approximated as H*sqrt(g/h), where H is wave height. Now here we just assume wave height of 1 m, but that this could be different. However, generally have found this restriction to work pretty well. Now overland we use a 1 m minimum depth (h). In doing so we have 1*sqrt(g/h) = sqrt(gh) = sqrt(g), i.e., a Froude number (=u/sqrt(gh)) of 1 is the restriction. Perhaps a Froude number of two could be a more conservative restriction for overland flow in which case setting H=2 would work well.

I'll close this ticket since the other two created (#25 & #26) are going to cover what was discussed here

Add CFL constraint for size function
Related to requests in #24.

Examples:

python
hfun = ...
hfun.add_contours(...)
...
hfun.add_cfl_constraint(cfl_upper=1 , cfl_lower=-np.inf, ref_vel=10, ref_dt=150, rate=0.1)
...

where instead of `ref_vel` one can use the estimates discussed in the issue referenced at the top

@TPCollings, can you please get the `feature/cflconstraint` branch and test the newly added functionality? I added some basic tests, but I didn't really test it on a large model. Please let me know if you see any issues so that I can fix before merging. Thanks!
Please see: URL and URL for the new methods. Also you can look at the tests added at URL

@WPringle if you have time at some point can you please take a look at the functions implemented here to see if everything seems right for CFL constraint:
 URL

Hi @SorooshMani-NOAA, I've got a workshop till friday and am away till Wednesday next week but will do it by the end of next week if that's okay? Thanks again for your help.

Cheers,

Tom

Hi @TPCollings, there's no rush for merge, please take your time :)

Hi Soroosh,

Sorry I still haven't had a chance to test this! Yeah probably best to
merge it and I can let you know if there's an issue. Cheers
---------------------------------------------------------------
*Thomas Collings* | *Fathom™*
Developer | www.fathom.global | @fathom_global

wrote:


Question: [Critical] Attackers may craft invalid domain bundle/execution receipt with excessive fraud proof size, making it impossible to submit a fraud proof on consensus chain?

There's also #2365. Can we have bundle limit in domain config instead of block limit? Especially since domain block limit is not enforced.
So in domain config we would have `target_bundles_per_slot` (1 or less, or more) and `max_bundle_weight`. Then the average domain block weight would be `target_bundles_per_slot * max_bundle_weight/SLOT_PROBABILITY` and each domain should have hardware requirements to be able to achieve that.

[Critical] Attackers may craft invalid domain bundle/execution receipt with excessive fraud proof size, making it impossible to submit a fraud proof on consensus chain
# Issue description

When a bundle/execution receipt with an invalid execution result is constructed, someone will have to generate a fraud proof for it and submit it by dispatching a `submit_fraud_proof` call on the consensus chain. For the case of an invalid extrinsic execution result, the fraud proof will be of type `InvalidStateTransitionProof` and this fraud proof will contain one or two storage proofs (`InvalidStateTransitionProof.proof` and `InvalidStateTransitionProof.execution_phase.ApplyExtrinsic.extrinsic_proof`).

If the failing extrinsic is doing a lot of storage access, this will lead to large storage proofs, possibly to an extent where the fraud proof exceeds the consensus chain `MAX_BLOCK_LENGTH = 5 MiB`. If this is the case, honest domain operators may know that the block is invalid but they can never successfully dispatch a `submit_fraud_proof` for it.

In order to limit the amount of storage changes, the 2D weight system should be used with an appropriate limit for the `proof_size` weight in order to ensure that it is always possible to submit a fraud proof when needed. As of now the `MAXIMUM_BLOCK_WEIGHT` is configured to `Weight::from_parts(u64::MAX, u64::MAX)`, so there is effectively no limit at all.

# Risk

If an invalid bundle/execution receipt cannot be discarded via a fraud proof due to this, it will eventually be considered final. If the storage root after executing the invalid bundle contains outgoing transfer messages minting SSC out of thin air, these transfers will be accepted by the consensus chain or sibling domains, effectively allowing an attacker to mint arbitrary amounts of SSC.

# Mitigation suggestion

Use an appropriate limit for the `proof_size` weight (with some safety margin to the consensus chain block length) to ensure that it is always possible to submit a fraud proof for an invalid bundle. Extra care needs to be taken to ensure that no calls can lead to a proof size larger than the predicted value from the weight calculation since this may directly lead to a situation where an invalid execution receipt is accepted since nobody can submit a fraud proof for it.

I think there might be an issue for this already and there was some progress done on this, which wasn't quite finished

This issue tracks the same case described here - URL

We partially solved it for using frontiers `POV` size ratio. Corresponding code is here - URL
This ensures on the EVM side the gas limit is split between storage access and compute. Currently set to `1/4` of the gas limit.
On the substrate end, since team writes the pallets, we de-prioritzed this until we have add benchmarks to each pallet used and come back if we there would be a scenario where this could happen.

Having said that, I'm curious if there is a scenario you have come up with

Can provide an update on the domain block weight limits? As of now we still have `MAXIMUM_BLOCK_WEIGHT: Weight = Weight::from_parts(u64::MAX, u64::MAX)` and without an actual limit it can't be guaranteed that a fraud proof will fit into the conensus chain block size limit.

@jakoblell We are aware of this. We will prioritse this change soon and post a follow-up pr with proper extrinsics limits. Thank you!

@vedhavyas is the issue not fixed between #2568 and #2651 by setting up bundle weight limits that in turn limit extrinsics?

@jakoblell
For substrate based chains, we ensure the total weight of all transactions in a given Bundle are at most BundleWeightLimit. More here - URL
This should not allow FP storage proofs to blow up since the all the reads are accounted in individual tx_weight.

One thing note here is that if the txn weights are outdated or wrong, this would be a problem since actual execution weight could be completely different. This may not be a problem for Runtimes we provide like EVM or AutoID but poses a bigger issue if and when we allow custom domain runtimes where they could completely give wrong weight for a given transaction.
In such case, this issue will come back where honest operator wont be able to submit a FP.

Ideal solution would be to get the actual execution weight for a given transaction and then use it in BundleWeightLimit. But getting such a would require us to execute the txn, collect the actual weight. This would be in-efficient but necessary unfortunately.

This might not be necessary if we get the actual execution weight but simply limiting the proof size will not solve the issue since proof sizes can go over anyway and fraud proof will not submitted due to size limits

URL

To me it looks like this is just in the domain block/bundle production code used by honest domain operators - but this weight limit isn't actually enforced on chain. A malicious domain operator can trivially patch out this check and submit an invalid and overweight domain block. This can lead to a situation where a fraud proof cannot be submitted due to the size of the required storage proof.

In order to prevent that, the domain runtime should directly reject blocks exceeding a reasonable weight limit (including a proof size weight). Enforcing a block weight limit on a substrate-based chain is typically implemented via `frame_system::CheckWeight` based on the `BlockWeights` configuration, which is currently still configured to `u64::MAX`:

* URL
* URL
* URL

Well there may be some gaps/inaccuracies in the proof size weighting but in theory it should be sufficient to limit the required size for the storage proof required in a fraud proof. In practice there should be some safety factor between the allowable domain block proof size weight and the consensus chain block length limit so that a fraud proof will always fit into the limit.

Correct. This should be handled with this Fraud proof - URL

We have set the Max limit for the Domain block because we have seen ExhaustResources during the Domain block import. Ideally, we want to ensure all the domain transactions in Consensus bundles should be executed on Domain. We are okay if the domain block import is slower but nonetheless executed. In order to avoid the excessive size coming from each extrinsics, we have introduced the POV size for EVM specifically and then we introduced MAX_BUNDLE_WEIGHT on consensus chain. One regression introduced with MAX domain Block limit is max extrinsic weight. We have an issue to fix the max_extrinsic_weight on domains while keeping the MAX domain block weight.

Once this above issue is fixed, there is reduced weight for each extrinsic and operators use that to calculate Domain Bundle weight. If the malicious operator do skip it, fraud proof will come into picture to slash that specific operator.

Let me know if that make sense @jakoblell

@jakoblell I have a PR up here - URL
That should fix the limits for proof size for domains

@jakoblell Can you confirm if this is still an issue post #2801

The `max_extrinsic` weight limit for domain extrinsic seems to be too large
The `max_extrinsic` weight limit seems to become too large since we set the max domain block weight to `u64::MAX` due to `max_extrinsic` is derived from `max_total`:
 URL

This may be an issue, especially for the evm transaction whose weight is computed from the `gas_limit` and it is provided by the user, if `gas_limit` is set to a large value the evm transaction will enter the operator's tx pool while it will fail to include in the bundle as it exceeds the bundle weight limit, thus it may stay at the tx pool forever.

cc @vedhavyas

True, I initially wanted to limit max_extrinsic weight to a sane value but somehow missed to update that in the previous PR. This was not straight forward since the BlockWeights max_block_weight was used to derive this. I think we need to manually create the Block weight unfortunately.

This is getting a bit more tricky since we introduced bundle limit in URL because the `max_extrinsic` weight needs to be smaller than the max bundle weight limit while the bundle limit is calculated from the domain config which is a param of the `instantiate_domain` call and is not a constant value. So different domain instances may need different `max_extrinsic` values, as a result, we may need to store the bundle limit in the domain state either via genesis state (like domain id) or inherent extrinsic (like XDM channel allow list).

Also, in gemini-3h, the bundle limit of domain 0 is now `DomainBundleLimit { max_bundle_size: 357469, max_bundle_weight: Weight { ref_time: 136363636363, proof_size: 1257732550480196701 } }`, meaning the `max_extrinsic` is ~136ms compare to previous 1500ms with domain block weight limit. cc @dariolina


Question: some warnning happened when i run manipulation.ipynb?

Closing, as it looks like this issue has been resolved. Please re-open if this is not the case. Thanks!

some warnning happened when i run manipulation.ipynb
/home/ubuntu/miniconda3/envs/mujoco_playground/lib/python3.10/site-packages/jax/_src/interpreters/xla.py:132: RuntimeWarning: overflow encountered in cast
 return np.asarray(x, dtypes.canonicalize_dtype(x.dtype))
/home/ubuntu/miniconda3/envs/mujoco_playground/lib/python3.10/site-packages/jax/_src/interpreters/xla.py:132: RuntimeWarning: overflow encountered in cast
 return np.asarray(x, dtypes.canonicalize_dtype(x.dtype))
/home/ubuntu/miniconda3/envs/mujoco_playground/lib/python3.10/site-packages/jax/_src/interpreters/xla.py:132: RuntimeWarning: overflow encountered in cast
 return np.asarray(x, dtypes.canonicalize_dtype(x.dtype))
2025-01-16 18:44:38.791223: E external/xla/xla/service/slow_operation_alarm.cc:73]
********************************
[Compiling module jit_generate_eval_unroll] Very slow compile? If you want to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.

@kassasin

I just ran the colab using a colab instance, and can't repro. Does everything train as expected or is JIT hanging for too long?

Yes， it has running two days！
I am using 4090 to run this tutorial .

Ok that's super interesting and also a bummer. I can't repro on 1x RTX 4090. This sounds like it's related to URL Will have to open a bug with the JAX team.

Can you try lowering `num_evals_envs` to something like 32 in the brax PPO call for now?

Hello, I still meet this problem.
My ppo_params is:

action_repeat: 1
batch_size: 512
discounting: 0.97
entropy_cost: 0.02
episode_length: 150
learning_rate: 0.001
network_factory:
 policy_hidden_layer_sizes: !!python/tuple
 - 32
 - 32
 - 32
 - 32
 policy_obs_key: state
 value_hidden_layer_sizes: !!python/tuple
 - 256
 - 256
 - 256
 - 256
 - 256
 value_obs_key: state
normalize_observations: true
num_envs: 32
num_evals: 4
num_minibatches: 32
num_timesteps: 20000000
num_updates_per_batch: 8
reward_scaling: 1.0
unroll_length: 10

Hi, I have the same problem. I ran it on Colab and my local RTX4090 and had the same problem. On my local RTX4090, there is one more warning about "Very slow compile".

action_repeat: 1
action_scale: 0.04
ctrl_dt: 0.02
episode_length: 150
reward_config:
 scales:
 box_target: 8.0
 gripper_box: 4.0
 no_floor_collision: 0.25
 robot_target_qpos: 0.3
sim_dt: 0.005

action_repeat: 1
batch_size: 512
discounting: 0.97
entropy_cost: 0.02
episode_length: 150
learning_rate: 0.001
network_factory:
 policy_hidden_layer_sizes: !!python/tuple
 - 32
 - 32
 - 32
 - 32
 policy_obs_key: state
 value_hidden_layer_sizes: !!python/tuple
 - 256
 - 256
 - 256
 - 256
 - 256
 value_obs_key: state
normalize_observations: true
num_envs: 2048
num_evals: 4
num_minibatches: 32
num_timesteps: 20000000
num_updates_per_batch: 8
reward_scaling: 1.0
unroll_length: 10

/home/ /anaconda3/envs/ mjc/lib/python3.13/site-packages/jax/_src/interpreters/xla.py:132: RuntimeWarning: overflow encountered in cast
 return np.asarray(x, dtypes.canonicalize_dtype(x.dtype))
2025-01-27 17:01:38.465476: E external/xla/xla/service/slow_operation_alarm.cc:73]
********************************
[Compiling module jit_generate_eval_unroll] Very slow compile? If you want to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
********************************

Hi @Elfits and @kassasin , I just tried on my RTX 4090 locally, and I cannot reproduce. I tried multiple environments (PandaRobotiqPushCube, PandaPickCubeOrientation, LeapCubeReorient).

Can y'all give a minimal reproducible example, and ideally open an issue here with the XLA dump: URL using latest JAX version?

Ok never mind! Was able to repro with `PandaRobotiqPushCube`. Will open a bug and post here.

Hi @Elfits that warning has been happening for some time and is mostly benign

another data point, I am experiencing the same issue when running `PandaRobotiqPushCube ` with RTX 4090.

Is this issue fixed? the same warninng when running PandaRobotiqPushCube with RTX 3060
`mujoco==3.2.7
jax==0.5.2
brax==0.12.1`

@yun-long @AvalonGuo if y'all are experience a slow operation alarm, please open a new bug with some details to reproduce!

 URL fixed the original issue a while ago and the "RuntimeWarning overflow" is benign, so I'm closing the issue for now

Thanks! It is really helpful. Should I need reinstall some packages?

Operation very slow to compile since jax 0.4.36
### Description

Hi folks,

We've been having slow compilation issues since jax 0.4.36 with some of our JAX code. The slow compilation (i.e. takes O(hours) to run instead of
 Install packages

pip install --upgrade jax[cuda] jaxlib
pip install --upgrade mujoco
pip install --upgrade mujoco_mjx
pip install --upgrade brax

mujoco==3.2.7
mujoco-mjx==3.2.7
brax==0.12.1

Run this Python code:

python
import functools
from mujoco_playground import registry
from mujoco_playground import wrapper
from mujoco_playground.config import manipulation_params
from brax.training.agents.ppo import train as ppo
from brax.training.agents.ppo import networks as ppo_networks

env_name = 'PandaRobotiqPushCube'
env = registry.load(env_name)
env_cfg = registry.get_default_config(env_name)

ppo_params = manipulation_params.brax_ppo_config(env_name)
ppo_training_params = dict(ppo_params)
network_factory = ppo_networks.make_ppo_networks
if "network_factory" in ppo_params:
 del ppo_training_params["network_factory"]
 network_factory = functools.partial(
 ppo_networks.make_ppo_networks,
 **ppo_params.network_factory
 )

train_fn = functools.partial(
 ppo.train, **dict(ppo_training_params),
 network_factory=network_factory,
)
make_inference_fn, params, metrics = train_fn(
 environment=env,
 wrap_env_fn=wrapper.wrap_for_brax_training,
)

The corresponding XLA dump is attached.

I also reran the same script with `num_evals=0` within `train_fn`, and the code runs fine (the slow compilation occurs somewhere here). I'm attaching both the working and non-working XLA dumps. We would really appreciate any help on this issue.

xla_dump_hanging_compilation.tar.gz
xla_dump_working.tar.gz

### System info (python version, jaxlib version, accelerator, etc.)

jax: 0.5.0
jaxlib: 0.5.0
numpy: 1.26.4
python: 3.12.3 (main, Sep 10 2024, 15:47:39) [GCC 13.2.0]
device info: NVIDIA GeForce RTX 4090-1, 1 local devices"
process_count: 1
platform: uname_result(system='Linux', node='btaba.mtv.corp.google.com', release='6.10.11-1rodete2-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1rodete2 (2024-10-16)', machine='x86_64')

$ nvidia-smi
Tue Jan 28 13:30:18 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.01 Driver Version: 535.216.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:61:00.0 Off | Off |
| 0% 50C P2 22W / 450W | 397MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2663746 C .../.pyenv/versions/mjx-312/bin/python 390MiB |
+---------------------------------------------------------------------------------------+

HLO reproducer: URL

bazel run --config=cuda //xla/tools:run_hlo_module -c opt -- --xla_disable_all_hlo_passes --input_format=hlo --random_init_input_literals --platform=CUDA /opt/repro.hlo

I'll take a look which pass is blowing up. Hopefully it's not in LLVM :-).

The quick hack did not work. I'll try to migrate us to the indexing maps next week.

Thanks @jreiffers and @pifon2a for taking a look, really appreciate it!

So, i have a new version of a computation partitioner that completely relies on indexing maps. That did not help and it even outlines the same number of functions. Disabling the inliner helps and it compiles quickly. I will check, what's happening there.

Ok, the issue was within the inliner itself. I will upload the fix today.

The fix makes compilation of one of the tests in JAX slow. Everything else became better... Looking.

Indexing map-based partitioner and Tweaked inliner fixed the issue. Let me know if you still have problems with this.


Question: Name of project/tool?

@ugurdogrusoz item 6 ( test.sbgn) is now working. Reason of error is that test.sbgn does not include mapProperties, so there is undefined errors in code, it is fixed by adding if else statement.

Name of project/tool
- [x] 1. Name the service as SyBValS (**Sy**stems **B**iology **Val**idation **S**ervice) everywhere in the sample application, including the About dialog
- [x] 2. Name this repository as "sybvals"
- [x] 3. Write a README introducing the service (use a format similar to this)

I went over the README, revised the options and examples in the Usage section. I also added small explanations to the first part that the sbgn file with corrected errors is produced. The only issue is that the small test file that I used in README works for validation query, but gives error we try to resolve errors. I added this to issue #8.

It would be good if we change the screenshots since now we are applying incremental layout between validation and error fix.

Errors in deployment
- [x] When I click on "Errors" download button when no file is uploaded, it throws an error in the console. It doesn't prevent the program from running, but let's fix it anyway.
- [x] After I resolve errors in Sample 1 & 2, download the SBGNML files and then try to open them in Newt, it throws an error in the console and the maps cannot be loaded.
- [x] When I tried to validate "Insulin-like growth factor (IGF) signaling" map from Newt samples, it gives error "Error detail: SyntaxError: Unexpected token '<', "".
- [x] When I tried to validate "Drosophila cell cycle" map from Newt samples, even though Newt shows 3 errors, sybvals shows 6 errors (I think it also adds the reverse version of each error) and doesn't highlight the errors. Additionally, when tried to resolve errors, it returns an error dialog.
- [x] vitamins_b6_activation_to_pyridoxal_phosphate.sbgn.txt When we validate this file, service finds 25 errors but error numbers start from 15 and go to 39. In addition, resolving errors give error.
- [x] For the examples given in README, I used test.sbgn.txt. it is a small file with only 3 nodes and two edges. When I test it, validation works correctly, but error resolving gives error.

Now, sample SBGNML files( after error resolving) successfully opened in Newt. I tried to click "Errors" download button when there is no file uploaded and I didn't get any error. Third error is related to #3. Reason of this error is "submap" is not handled in stylesheet.

In item 1, I still see the mentioned error.
I also added a new item (item 4).

I see that only item 1 is resolved. I ignored item 3, but there is no change in item 4.

For item 4, we now show 3 errors and highlight them, it is ok. But when I try to resolve errors in
drosophila_cell_cycle.nwt.txt it works but if I try drosophila_cell_cycle.sbgn.txt it gives error. I generated the first file with save in Newt and the second one by exporting as SBGN-ML Plain. Please remove .txt from the extensions to test the files.

@hasanbalci item5( vitamin b6) is working now. It is related to listing pd10102 errors.

it is working for both .sbgn and .nwt files now. @hasanbalci

@YusufZiyaOzgul Thanks, droshophila example works now. However, after we resolve its errors, if we download the SBGN file and try to validate again, it finds a new error with an element unrelated to the previous ones. This doesn't happen in Newt.


Question: Version 1.10+ no longer finds checkpoints in custom install folder.

Okay, but the issue is that SD1.5 checkpoints are detected as "stab" for some reason.
You said you updated ComfyUI and custom nodes, but what are the exact versions of ComfyUI and comfyui-tooling-nodes you are using?
I have both on latest (Comfy: a252963f956a7d76344e3f0ce24b1047480a25af, comfyui-tooling-nodes: b2496a3f132f8c3f7d452a0960c422f55c33d128) and can't reproduce this...

Version 1.10+ no longer finds checkpoints in custom install folder.
Error: _The checkpoint used by this style is not installed_.

My ComfyUI install is a Windows portable version with custom .yaml paths. It had been working flawlessly with the plugin until 1.10/11.
Updating caused missing Python errors likely caused by ComfyUI's portable version Python folder being called _python embeded_, while the Krita plugin searches for _python_ instead. Although this had been working and is unrelated.

Now it can only detect SDXL models and no 1.5 models. If I remove the custom checkpoint path from the .yaml file it no longer detects any checkpoints at all. Every other folder path in the .yaml (LoRAs, etc.) is detected except for checkpoints. ComfyUI itself can read the checkpoint folder.

That was intentional, the comfy-managed-by-plugin option isn't meant to manage a custom install (bad things can happen, use at your own risk)

Maybe the same as #267, make sure you have the latest version of comfyui-tooling-nodes

Just as you responded I saw that someone had brought this issue up before, oops!
The files in the link you posted fixed the problem.

Missing SD 1.5 workflow even with all the models installed.
Capture
It's been like this since I updated ComfyUI nightly along with its prerequisites and custom nodes to their latest version.

Only SDXL workflows work, SD 1.5 is missing "Stable Diffusion Checkpoint" even though I've got the same sd 1.5 based checkpoints in the same directory as usual.

I thought maybe ComfyUI update broke the checkpoint base model detection somehow, but at least for me it's still working with latest version.
Can you check the output of URL
Are your checkpoints listed? And is the base model detected correctly ("sd15" if it's SD1.5 checkpoint)?

yes every model is listed there in etn/model_info but as base_model: "stab"

ive pulled everything to the latest commit again just now, i tend to do this every couple of days/a week, its still detecting sd 1.5 as "stab", could it be a python dependency issue?

No, my guess is still that comfyui-tooling-nodes is outdated (for some reason).

Reason: previous versions just used the first 4 letters lower case of comfy's name for a base model. Recently comfy added support for `Stable_Zero123` base model, and your SD1.5 checkpoints are probably being misdetected as that.

But I just don't see how the current code would ever return "stab", even if the detection didn't work. It no longer does the first-4-letters thing...

ive deleted comfyui-tooling-nodes and manually downloaded instead of pulling or cloning it, now it works, bases_model detects as sd15

thanks


Question: Improve gas estimation for create transactions?

Hi all,

I've tried reproducing this on Optimism, Optimism Sepolia, Arbitrum and Arbitrum Sepolia on both Anvil forks and actual deployments and am unable to reproduce it.

On Arbitrum and Arbitrum Sepolia (live) we flag it as an exception, estimate via RPC and trigger slow mode.

See: URL

Given that the issue does not occur anymore or cannot be reproduced reliably I'm marking this as `resolved`.

The existing workaround of `--with-gas-multiplier` exists if one happens to run into this issue.

Improve gas estimation for create transactions
### Component

Forge

### Describe the feature you would like

followup for URL

- [ ] estimate gas for `executor.deploy`

### Additional context

_No response_

@mattsse What did you have in mind here? Is this related to item 3 in URL Unsure if we should keep this open and track it in #4444

Marking as `resolved` as I think it is likely resolved or, if not, will be handled by other tickets detailing specific issues with individual chains or configurations

Feel free to re-open if this is still an issue we should address

bug(`forge script`): tx succeeds during simulation but fails on any regular function call that occurs in a script after contract creation
### Component

Anvil

### Have you ensured that all of these are up to date?

- [X] Foundry
- [X] Foundryup

### What version of Foundry are you on?

forge 0.2.0 (0962fd3 2022-06-16T18:19:05.497315Z)

### What command(s) is the bug in?

_No response_

### Operating System

macOS (Apple Silicon)

### Describe the bug

Executing a contract func that updates storage on a contract that doesn't have a constructor fails on a forked instance of optimism or arbitrum (using anvil). On mainnet and polygon forks, there's no issue.

solidity
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

contract Contract {
 address public test;

 function initialize(address _test) external {
 test = _test;
 }
}

solidity
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import "forge-std/Script.sol";

import {Contract} from "src/Contract.sol";

contract ContractScript is Script {

 Contract public testContract;

 function run() public {
 vm.broadcast();
 testContract = new Contract();

 vm.broadcast();
 testContract.initialize(address(1));
 }
}

What's interesting is the simulation is successful, but the actual broadcast tx fails:

$ forge script script/Contract.s.sol --rpc-url " URL --private-key $ANVIL_ACCT_9 --broadcast -vvvv [14:41:16]
[⠒] Compiling...
Nothing to compile
Traces:
 [93261] ContractScript::run()
 ├─ [0] VM::broadcast()
 │ └─ ← ()
 ├─ [49699] → new Contract@0x700b6a60ce7eaaea56f065753d8dcb9653dbad35
 │ └─ ← 248 bytes of code
 ├─ [0] VM::broadcast()
 │ └─ ← ()
 ├─ [2502] Contract::initialize(0x0000000000000000000000000000000000000000)
 │ └─ ← ()
 └─ ← ()

Script ran successfully.
Gas used: 93261
==========================
Simulated On-chain Traces:

 [106971] → new Contract@0x700b6a60ce7eaaea56f065753d8dcb9653dbad35
 └─ ← 248 bytes of code

 [23694] Contract::initialize(0x0000000000000000000000000000000000000000)
 └─ ← ()

==========================

Estimated total gas used for script: 131151

Amount required: 0.000262302 ETH

==========================

###
Finding wallets for all the necessary addresses...
##
Sending transactions [0 - 1].
⠉ [00:00:00] [#################################################################################################################################################################################################################################################] 2/2 txes (0.0s)
Transactions saved to: broadcast/Contract.s.sol/10/run-latest.json

##
Waiting for receipts.
⠙ [00:00:07] [#############################################################################################################################################################################################################################################] 2/2 receipts (0.0s)
#####
✅ Hash: 0x616898701db8efc33e972bfb36467311b28cfc49ed863f70dfe2f4a1d48a4c71
Contract Address: 0x700b6a60ce7eaaea56f065753d8dcb9653dbad35
Block: 12064970
Paid: 0.000213942 ETH (106971 gas * 2 gwei)

#####
❌ Hash: 0x03db02d9ff00fb3f4e1c6550670aa6c57e8a9361d1e406a34e391e0de5ee0b21
Block: 12064971
Paid: 0.0000469 ETH (23450 gas * 2 gwei)

Transactions saved to: broadcast/Contract.s.sol/10/run-latest.json

Error:
 0: ["Transaction Failure: 0x03db…0b21"]

Location:
 cli/src/cmd/forge/script/receipts.rs:75

If you remove logic that updates any storage state, the tx succeeds.

I just gave the script I put above a try on prod optimism, the tx for the initialize call fails there as well:

$ forge script script/Contract.s.sol --rpc-url $OPTIMISM_RPC_URL --private-key --broadcast --slow -vvvv [16:33:43]
[⠢] Compiling...
Nothing to compile
Traces:
 [113155] ContractScript::run()
 ├─ [0] VM::broadcast()
 │ └─ ← ()
 ├─ [49699] → new Contract@0x8eb4600bbab2286a46103f4923866de9653b737b
 │ └─ ← 248 bytes of code
 ├─ [0] VM::broadcast()
 │ └─ ← ()
 ├─ [22402] Contract::initialize(0x0000000000000000000000000000000000000001)
 │ └─ ← ()
 └─ ← ()

Script ran successfully.
Gas used: 113155
==========================
Simulated On-chain Traces:

 [106971] → new Contract@0x8eb4600bbab2286a46103f4923866de9653b737b
 └─ ← 248 bytes of code

 [43606] Contract::initialize(0x0000000000000000000000000000000000000001)
 └─ ← ()

==========================

Estimated total gas used for script: 128175

Amount required: 0.000000128175 ETH

==========================

###
Finding wallets for all the necessary addresses...
##
Sending transactions [0 - 1].
⠁ [00:00:00] [########################################################################################################################>------------------------------------------------------------------------------------------------------------------------] 1/2 txes (0.4s)⠉ [00:00:07] [#############################################################################################################################################################################################################################################] 1/1 receipts (0.0s)
#####
✅ Hash: 0x1a972cf266cbfd61a028e1fdd694686130ce4b6dc0c7202bd21212ef775c8589
Contract Address: 0x8eb4600bbab2286a46103f4923866de9653b737b
Block: 12073464
Gas Used: 106971

⠉ [00:00:07] [#################################################################################################################################################################################################################################################] 2/2 txes (0.0s)⠉ [00:00:07] [#############################################################################################################################################################################################################################################] 1/1 receipts (0.0s)
#####
❌ Hash: 0x1b07ca2cce8b7302dbdb0e4a257c07d19689647c50de8df8a3549a2eedd1a3ff
Block: 12073469
Gas Used: 21204

Transactions saved to: broadcast/Contract.s.sol/10/run-latest.json

Error:
 0: ["Transaction Failure: 0x1b07…a3ff"]

The failed tx.

But, notice that executing a tx with the same args but with `cast` and a gas limit of `75000` results in success:

$ cast send 0x8eb4600bbab2286a46103f4923866de9653b737b "initialize(address)" 0x0000000000000000000000000000000000000001 --gas 75000 --rpc-url $OPTIMISM_RPC_URL --private-key [16:41:41]

blockHash 0xf6c411ec971b21c1ec5c41487929b814bad5be151f4831de7b07d0e36c6a44c0
blockNumber 12074683
contractAddress
cumulativeGasUsed 43606
effectiveGasPrice
gasUsed 43606
logs []
logsBloom 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
root
status 1
transactionHash 0x1493c477c18629ab75f6759f95bed118517f6a4e66602460a2c3643c93486801
transactionIndex 0
type

So, seems like this is a gas estimation issue

Per @mds1 suggestion let's go with the following 2 steps:

1 Add a broadcast overload that lets you specify the gas limit for the tx, `vm.broadcast(uint256 gasLimit)` and `vm.broadcast(address sender, uint256 gasLimit)`
2. add bespoke logic for each L2, determined by chainId, this will need to be fork-aware when #1715 is merged

cc @joshieDo

We should probably add those broadcast interfaces. However, this is actually a result from another issue.

Since L2s have different gas calculations, we discard the gas estimated locally, and request a new one from the RPC. To support batching, they're happening at the same time for tx1 and tx2. However, tx2 needs tx1 to be submitted and accepted first.

We should probably force `--slow` on L2s or any other transaction that requires `--slow`. In this case, even with `--slow` won't work, since we estimate everything before we start sending, which needs to be fixed.

I was under the impression Arbitrum and Optimism had different gas metering, but it seems that only Arbitrum has.

Just summarizing what we ended up on as the path forward:
1. Remove Optimism from the `has_different_gas_calc` function

 URL
 URL

2. Force `--slow` on Arbitrum and Optimism. Both of don't have mempools like L1's so if you send multiple txs they get rejected with `nonce too low`.

3. Move RPC gas estimation right before sending the intended tx, and add a note to the output indicating the "total gas/ETH cost estimates" shown beforehand aren't guaranteed to be accurate (since we don't estimate Optimism costs correctly)

4. This one wasn't settled on, but I think we should change the broadcast overloads to specify margin on the gas estimate, instead of absolute limits. So if I'm estimating gas for a uniswap v2 trade where cost varies by block position, I can do `vm.broadcast(20)` to mean "add 20% margin to the gas estimate". IMO this is better UX than needing to harcode gas limits directly.

For accurate fee estimation on Optimism, the L2 (execution) fee and the L1 (availability) fee needs to be taken into account. See here for how to do so. If special logic is added that is chain aware, perhaps handling this could just be abstracted away from users. See URL for a higher level explanation

Of the 4 items listed in my above comment, I believe 1 and 2 are implemented, 3 and 4 are not.

Just a small note on all of this, recently stumped me and had to use the `-g` flag.

Could we also get a better error message if the transaction runs out of gas? `Transaction Failure: 0x1b07…a3ff` is not the most obvious. I'm down to implement this myself, just curious if others would agree or there's some reason we don't currently do that.

+1 on @devanonon's comment, haven't gotten around to opening an issue yet but here's a related repro with other issues: URL

This issue still exists, and I believe it applies to any regular function call that occurs in a script after contract creation (i.e. not necessarily limited to an initialization call). I used `--with-gas-multiplier` to resolve it.

Optimistically marking this ticket as resolved. We've since moved to Alloy and URL indicated a similar issue has been resolved.

Feel free to re-open if there are still outstanding issues or alternatively open a new ticket that is specific to your situation.

Can confirm this is still an issue. My instance occurred while running a script aimed at optimism sepolia. I second that it does apply to any regular function call that occurs in a script after contract creation as @simplyoptimistic offered. I also used `--with-gas-multiplier` to resolve the issue.


Question: Resetting Rewards doesn't reset Ads count?

Verification in progress with

Brave | 1.76.74 Chromium: 134.0.6998.89 (Official Build) (x86_64)
-- | --
Revision | f529926c39ddc08b1e0873f4b8befdca4984b492
OS | macOS Version 14.7.4 (Build 23H420)

Resetting Rewards doesn't reset Ads count
### Description

While verifying the CR 134 manual pass, noticed resetting rewards doesn't reset ad count. Except for Ads data, other data is reset/wiped out. Tried few times and was able to reproduce every time. However, I could not reproduce in 1.75.x. Reproduced in 1.76.63.

### Steps to reproduce

1. Install 1.77.52
2. launch Brave
3. enabled Rewards
4. connected uphold
5. contributed BAT via rewards panel
6. waited for few ads
7. opened brave://rewards
8. clicked ... menu
9. clicked `Disable and reset Rewards`
10. Rejoined Rewards

### Actual result

The ads count still shown

 URL

### Expected result

Ad count from before resetting rewards should not be shown

### Reproduces how often

Easily reproduced

### Brave version (brave://version info)

Brave | 1.77.52 Chromium: 134.0.6998.15 (Official Build) nightly (64-bit)
-- | --
Revision | d2b1e97cd8ae806f9aa05a0204a9b953536c5bce
OS | Windows 11 Version 24H2 (Build 26100.2894)

### Channel information

- [ ] release (stable)
- [x] beta
- [x] nightly

### Reproducibility

- [ ] with Brave Shields disabled
- [ ] with Brave Rewards disabled
- [ ] in the latest version of Chrome

### Miscellaneous information

@Miyayes @zenparsing
cc: @brave/qa-team

I could not reproduce using `1.77.55`. @zenparsing mentioned maybe some kind of race condition?

 URL

Just adding some context re: running into the above issue when I asked @Miyayes to help me change my desktop `Nightly` installation from `Uphold` -> `Solana` on chain. Once I restarted my `Rewards`, everything was cleared/disconnected from Uphold other than the ads that I viewed. They were still listed as ~160 ads seen this month even though `rewards` was restarted. I connected my Solana wallet without any issues but the ads never cleared. So I'm currently at ~180 ads. So it definitely wasn't cleared when I reset `rewards`. Basically what @GeetaSarvadnya reported above.

On the other hand, when I did the same thing on `Android` re: restarting my `rewards` to get a new `Payment ID` to get whitelisted, it cleared the ads count without any issues.

The issue is reproducible on

Brave | 1.77.61 Chromium: 134.0.6998.24 (Official Build) nightly (64-bit)
-- | --
Revision | f5bab96d17b03e45c0253dc2b822f4706ff2739b
OS | Windows 10 Version 22H2 (Build 19045.5487)

Image

<

This issue seems to be reproducible on Windows but not MacOS.

On Windows, I saw this intermittently in `Rewards.log` immediately after resetting.

[Feb 25, 2025 3:24:07.3 PM GMT:INFO:account.cc(204)] Initialize confirmations
[Feb 25, 2025 3:24:07.3 PM GMT:INFO:ads_impl.cc(69)] Initializing ads
[Feb 25, 2025 3:24:07.6 PM GMT:ERROR:database_manager.cc(88)] Failed to create or open database
[Feb 25, 2025 3:24:07.6 PM GMT:ERROR:ads_impl.cc(417)] Failed to create or open database
[Feb 25, 2025 3:24:07.6 PM GMT:ERROR:ads_impl.cc(430)] Failed to initialize ads

This occurs using both the new and old Rewards pages.

The above requires `1.76.74` or higher for `1.76.x` verification 👍

Verification **PASSED** on

Brave | 1.76.74 Chromium: 134.0.6998.89 (Official Build) (64-bit)
-- | --
Revision | f529926c39ddc08b1e0873f4b8befdca4984b492
OS | Windows 10 Version 22H2 (Build 19045.5555)

- Verified the test plan from URL
- Verified the Test pan via URL

 Case 1: View ads in non-connected state and reset rewards_PASSED

1. Clean profile `1.78.18`
2. Enable rewards
3. View couple of ads
4. Confirmed that ads count is shown on the brave://rewards page
5. Reset rewards via `more...`
6. Re-enable rewards
7. Confirmed that ads counter is reset to Zero

Ads count | Reset rewards | Re-enable rewards | Ads count reset to zero
-------------|-----------------|---------------------|----------------------
image | image | image | image

 Case 2: View ads in connected state and reset rewards_PASSED

1. Clean profile `1.78.18`
2. Enable rewards
3. View couple of ads
4. Confirmed that ads count is shown on the brave://rewards page
5. Reset rewards via `more...`
6. Re-enable rewards
7. Confirmed that ads counter is reset to Zero

Ads count | Reset rewards | Re-enable rewards | Ads count reset to zero
-------------|-----------------|---------------------|----------------------
image | image | image | image

 Case 3: Upgrade from broken profile_PASSED

1. Clean profile `1.76.61`
2. Enable rewards and connect to a custodian
3. View couple of ads
4. Confirmed that ads count is shown on the brave://rewards page
5. Reset rewards via `more...`
6. Re-enable rewards
7. Confirmed that ads counter is NOT reset to Zero

 `1.76.61` - Ads count | `1.76.61`- Reset rewards | `1.76.61` - Re-enable rewards | `1.76.61` - Ads count NOT reset to zero
-------------|-----|------|-----
image | image | image | image

8. Upgrade the profile to `1.78.18`
9. Confirmed that ads count is shown on the brave://rewards page
10. Reset rewards via `more...`
11. Re-enable rewards
12. Confirmed that ads counter is reset to Zero as expected

 `1.78.18` - Ads count | `1.78.18`- Reset rewards | `1.78.18` - Re-enable rewards | `1.78.18` - Ads count reset to zero
-------------|-----|------|-----
image | image | image | image

[ads] Crash `brave_ads::GetNextPaymentDate`
### Description

[ 00 ] brave_ads::GetAdsClient() ( immediate_crash.h:188 )
[ 01 ] brave_ads::GetProfileTimePref(std::__Cr::basic_string , std::__Cr::allocator > const&) ( pref_util.cc:110 )
[ 02 ] brave_ads::GetNextPaymentDate(std::__Cr::vector > const&) ( statement_util.cc:41 )
[ 03 ] brave_ads::BuildStatement(base::OnceCallback )>)::$_0::operator()(base::OnceCallback )>, bool, std::__Cr::vector > const&) const ( statement.cc:48 )
[ 04 ] void base::internal::DecayedFunctorTraits )>)::$_0, base::OnceCallback )>&&>::Invoke )>)::$_0, base::OnceCallback )>, bool, std::__Cr::vector > const&>(brave_ads::BuildStatement(base::OnceCallback )>)::$_0&&, base::OnceCallback )>&&, bool&&, std::__Cr::vector > const&) ( bind_internal.h:647 )
[ 05 ] void base::internal::InvokeHelper )>)::$_0&&, base::OnceCallback )>&&>, void, 0ul>::MakeItSo )>)::$_0, std::__Cr::tuple )>>, bool, std::__Cr::vector > const&>(brave_ads::BuildStatement(base::OnceCallback )>)::$_0&&, std::__Cr::tuple )>>&&, bool&&, std::__Cr::vector > const&) ( bind_internal.h:921 )
[ 06 ] void base::internal::Invoker )>)::$_0&&, base::OnceCallback )>&&>, base::internal::BindState )>)::$_0, base::OnceCallback )>>, void (bool, std::__Cr::vector > const&)>::RunImpl )>)::$_0, std::__Cr::tuple )>>, 0ul>(brave_ads::BuildStatement(base::OnceCallback )>)::$_0&&, std::__Cr::tuple )>>&&, std::__Cr::integer_sequence , bool&&, std::__Cr::vector > const&) ( bind_internal.h:1058 )
[ 07 ] base::internal::Invoker )>)::$_0&&, base::OnceCallback )>&&>, base::internal::BindState )>)::$_0, base::OnceCallback )>>, void (bool, std::__Cr::vector > const&)>::RunOnce(base::internal::BindStateBase*, bool, std::__Cr::vector > const&) ( bind_internal.h:971 )
[ 08 ] base::OnceCallback ::Run(bool) && ( callback.h:156 )
[ 09 ] content::RenderFrameHostImpl::RunModalConfirmDialog(std::__Cr::basic_string , std::__Cr::allocator > const&, bool, base::OnceCallback )::$_0::operator()(base::OnceCallback , bool, std::__Cr::basic_string , std::__Cr::allocator > const&) const ( render_frame_host_impl.cc:6468 )

Verification INPROGRESS on

Brave | 1.76.74 Chromium: 134.0.6998.89 (Official Build) (64-bit)
-- | --
Revision | f529926c39ddc08b1e0873f4b8befdca4984b492
OS | Windows 10 Version 22H2 (Build 19045.5555)

 Test case 1 - PASSED

* Start browser with fresh install
 EXPECTATION: `Brave Ads` is successfully started.
* Open search.brave.software and search for `giraffe`

* Click a search result ad
 EXPECTATION: Conversion URL patterns was added on brave://ads-internals page

* Click `Clear Ads Data` button
 EXPECTATION: Brave Ads conversion URL patterns list becomes empty. Browser didn't crash.

* Close the browser
 EXPECTATION: Browser didn't crash.

 Test case 2 - PASSED

* Start browser with fresh install
* Join Brave Rewards
 EXPECTATION: `Brave Ads` is successfully started.

[14967:259:0310/133845.006751:VERBOSE1:ads_impl.cc(75)] Initializing ads
[14967:259:0310/133845.061081:VERBOSE1:database_manager.cc(214)] Database is up to date on schema version 49
[14967:259:0310/133845.061586:VERBOSE1:database_maintenance.cc(49)] Scheduled database maintenance in 0 hours, 1 minute, 0 seconds at 13:39:45.061
[14967:259:0310/133845.061707:VERBOSE3:client_state_manager.cc(37)] Loading client state
[14967:259:0310/133845.061726:VERBOSE3:ad_events_database_table_util.cc(32)] Successfully purged all orphaned ad events
[14967:259:0310/133845.062150:VERBOSE3:client_state_manager.cc(129)] Successfully loaded client state
[14967:259:0310/133845.062202:VERBOSE3:confirmation_state_manager.cc(40)] Loading confirmation state
[14967:259:0310/133845.063042:VERBOSE3:confirmation_state_manager.cc(66)] Successfully loaded confirmation state
[14967:259:0310/133845.063068:VERBOSE1:ads_impl.cc(446)] Successfully initialized ads

* View any Brave ad
* Open brave://rewards page
 EXPECTATION: Ads count is more then 0

* Click `...` and `Disable and reset Rewards` button
 EXPECTATION: Brave Rewards was reset. Browser didn't crash.

* Close the browser
 EXPECTATION: Browser didn't crash.

Verified with

Brave | 1.76.74 Chromium: 134.0.6998.89 (Official Build) (64-bit)
-- | --
Revision | f529926c39ddc08b1e0873f4b8befdca4984b492
OS | Linux

 Test case 1 - PASSED

* Start browser with fresh install
 EXPECTATION: `Brave Ads` is successfully started.

[71422:71422:0311/115836.573213:VERBOSE1:ads_impl.cc(424)] Successfully initialized ads

* Open search.brave.software and search for `giraffe`

Image

* Click a search result ad
 EXPECTATION: Conversion URL patterns was added on brave://ads-internals page

Image

* Click `Clear Ads Data` button
 EXPECTATION: Brave Ads conversion URL patterns list becomes empty. Browser didn't crash.

Image

* Close the browser
 EXPECTATION: Browser didn't crash.

 Test case 2 - PASSED

* Start browser with fresh install
* Join Brave Rewards
 EXPECTATION: `Brave Ads` is successfully started.

[72503:72503:0311/121454.386392:VERBOSE1:ads_impl.cc(424)] Successfully initialized ads

* View any Brave ad
* Open brave://rewards page
 EXPECTATION: Ads count is more then 0

Image

* Click `...` and `Disable and reset Rewards` button
 EXPECTATION: Brave Rewards was reset. Browser didn't crash.

Image|Image
--|--

* Close the browser
 EXPECTATION: Browser didn't crash.

Verification **PASSED** on

Brave | 1.76.74 Chromium: 134.0.6998.89 (Official Build) (arm64)
-- | --
Revision | f529926c39ddc08b1e0873f4b8befdca4984b492
OS | macOS Version 15.3.1 (Build 24D70)

 Test case 1 - PASSED

* Start browser with fresh install
 EXPECTATION: `Brave Ads` is successfully started.
* Open search.brave.software and search for `giraffe`

* Click a search result ad
 EXPECTATION: Conversion URL patterns was added on brave://ads-internals page

* Click `Clear Ads Data` button
 EXPECTATION: Brave Ads conversion URL patterns list becomes empty. Browser didn't crash.

* Close the browser
 EXPECTATION: Browser didn't crash.

 Test case 2 - PASSED

* Start browser with fresh install
* Join Brave Rewards
 EXPECTATION: `Brave Ads` is successfully started.

[2256:259:0311/171918.859798:VERBOSE1:ads_impl.cc(68)] Initializing ads
[2256:259:0311/171918.971162:VERBOSE1:database_manager.cc(146)] Create database for schema version 45
[2256:259:0311/171918.978499:VERBOSE1:database_manager.cc(172)] Created database for schema version 45
[2256:259:0311/171918.979876:VERBOSE1:database_maintenance.cc(49)] Scheduled database maintenance in 0 hours, 1 minute, 0 seconds at 17:20:18.978
[2256:259:0311/171918.982516:VERBOSE3:ad_events_database_table_util.cc(32)] Successfully purged all orphaned ad events
[2256:259:0311/171918.983326:VERBOSE3:client_state_manager.cc(37)] Loading client state
[2256:259:0311/171918.983500:VERBOSE3:client_state_manager.cc(116)] Client state does not exist, creating default state
[2256:259:0311/171918.985724:VERBOSE3:confirmation_state_manager.cc(40)] Loading confirmation state
[2256:259:0311/171918.985837:VERBOSE3:confirmation_state_manager.cc(54)] Confirmation state does not exist, creating default state
[2256:259:0311/171918.985858:VERBOSE1:ads_impl.cc(424)] Successfully initialized ads

* View any Brave ad
* Open brave://rewards page
 EXPECTATION: Ads count is more then 0

Image

* Click `...` and `Disable and reset Rewards` button
 EXPECTATION: Brave Rewards was reset. Browser didn't crash.

* Close the browser
 EXPECTATION: Browser didn't crash.

Verification passed on Brave v1.76.74 on Lenovo TB-8506FS (Android 11.0)

#### Test case 3 (Android)

Start browser with fresh install
EXPECTATION: Brave Ads is successfully started.

16:19:03.287 V [VERBOSE1:ads_impl.cc(424)] Successfully initialized ads

Open search.brave.software and search for giraffe

Image

Click a search result ad
Open brave://ads-internals page
EXPECTATION: Conversion URL patterns was added on brave://ads-internals page

Image

Click Clear Ads Data button
EXPECTATION: Brave Ads conversion URL patterns list becomes empty. Browser didn't crash.

Image

Join Brave Rewards
EXPECTATION: Brave Ads is successfully started.

View any Brave ad
Open brave://rewards page
EXPECTATION: Ads count is more then 0

Image

Tap Reset button
EXPECTATION: Brave Rewards was reset. Browser didn't crash.

Image

Verification passed on 1.76.76 on iPhone 13 Pro Max (iOS 18.3.1)

#### Test case 4 (iOS)

Start browser with fresh install
EXPECTATION: Brave Ads is successfully started.

info 13:21:37.416675+0100 Client [ads] Successfully initialized ads

Open search.brave.software and search for giraffe
Click a search result ad
Open brave://ads-internals page
EXPECTATION: Conversion URL patterns was added on brave://ads-internals page

Image

Click Clear Ads Data button
EXPECTATION: Brave Ads conversion URL patterns list becomes empty. Browser didn't crash.

Image


Question: Question: How many threads can there be by default?

I'm closing this for now, I've done a bit of research and running the tests and monitoring using `netstat -a` didn't show any obvious problems here.

Will reopen when new information emerges or when a problem is reported by a user.

Question: How many threads can there be by default?
Hi
Does RestAssured have a limit on the number of threads?
If so, how is it installed and what is the default?

Also a question about the maximum waiting time for a request to be completed.
What is it like? How can I set a different value?

There is one more question
Can I see the response size in kilobytes?

Hey @workmichsem,

regarding the maximum number of connections: since I'm using `Sytem.Net.Http.HttpClient` under the hood, this is probably the default value for `System.Net.Http.HttpClientHandler.MaxConnectionsPerServer`, which defaults to `int.MaxValue`: URL

This might change due to #107, but I haven't had the time to work on that yet.

The maximum waiting time, or rather the `HttpClient` timeout, defaults to 100 seconds, but can be changed: URL

The response size in bytes can typically be found in the `Content-Length` header of the response. You can write a verification against it: URL (this uses the actual response body length, by the way, not the value of the `Content-Length` header). You can also calculate the length yourself by extracting the response body as a `string` and calculating its value. See also #92.

Hope that helps, if not, feel free to ask :)

Thank you very much for your answers!

Regarding the number of threads. So there is no way to change this value now? If not, will it be possible in the future?

No, there is no way to change this at the moment. What is your use case for this? It shouldn't be too hard to add a method to configure this, but I'm curious to see why you would need it.

Also, once I start working on #107 this might become possible, too.

Just trying to understand what the limitations are
Thank you very much for the information

No problem!

I just remembered something: you _can_ configure the maximum number of connections yourself by injecting an `HttpClient` of your own and configuring the `MaxConnectionsPerServer` property on the `HttpClientHandler` associated with the client.

See URL

If this is enough information for you, please close the issue. If not, leave it open and ask away :)

See if there’s a more efficient way of dealing with HttpClient
See URL for more information.

First step would be to see if HttpClientFactory provides a solution to this.


Question: Deploying from `x86_64-linux` to `aarch64-linux`?

Oh, AFAICS, build failure happens during checks build, which indeed needs target system capability for building the profile locally as a part of the check. A solution to this is to provide `--skip-checks` option

Deploying from `x86_64-linux` to `aarch64-linux`
Hello,

I would like to deploy updates to my Raspberry router using Deploy-rs. A work-in-progress commit has been made in my config, you can find it here.

When I run the command: `deploy .#router` (_`router` is my raspberry machine_), it downloaded ~1.2Gb and then, I got this issue.

$ deploy .#router
🚀 ℹ [deploy] [INFO] Running checks for flake in .
warning: unknown flake output 'deploy'
🚀 ℹ [deploy] [INFO] Evaluating flake in .
🚀 ℹ [deploy] [INFO] The following profiles are going to be deployed:
[router.system]
user = "root"
ssh_user = "pol"
path = "/nix/store/xlz3rd2dfkya33rjr1q6l1j849mh4lyw-activatable-nixos-system-router-23.11.20230607.381e92a"
hostname = "router"
ssh_opts = []

🚀 ℹ [deploy] [INFO] Building profile `system` for node `router`
error: a 'aarch64-linux' with features {} is required to build '/nix/store/c6mljancr1wmnc0vqgwwwi8ic86krix4-builder.pl.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
🚀 ❌ [deploy] [ERROR] Failed to push profile: Nix build command resulted in a bad exit code: Some(1)

Could you please help me by telling me if the `deploy-rs` has been correctly done (based on the commit) and if you have a solution on the issue I currently have ?

Thanks in advance.

Hi! Looks similar to URL
Currently, there are two solutions: either enable aarch64 cross-compilation on a machine you're deploying from (by setting `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`), or using `remoteBuild = true;` in case your target machine is powerful enough

@rvem Can we use crossSystem(instead of emulatedSystems) here?

Anyone got a working example on how to do cross builds and deploys?
I am looking for all combination of:

**Host**:
x86_64-linux
aarch64-darwin

**Target**:
aarch64-linux
x86_64-linux

@tcurdt The solution I found is switch to colmena, which supports custom `nixpkgs` instance by its `meta.nixpkgs` & `meta.nodeNixpkgs` parameters.

Aarch64 cross compilation errors
I'm trying to deploy some configurations to Aarch64 nodes from an X86_64 node running Linux.

I managed to cross compile my configuration without issued but it failed with the following error:

🚀 ℹ [deploy] [INFO] Activating profile `system` for node `nixpi`
[sudo] password for pi:
/nix/store/kwic2s1xgzl3j7spjvqsl0xswaa9p2w2-activatable-nixos-system-nixpi-23.05.20230412.fe2ecaf/activate-rs: line 2: /nix/store/5zcrj0jp36ckjbxw5j4nj6b18nzid5rv-deploy-rs-0.1.0/bin/activate: cannot execute binary file: Exec format error
/nix/store/kwic2s1xgzl3j7spjvqsl0xswaa9p2w2-activatable-nixos-system-nixpi-23.05.20230412.fe2ecaf/activate-rs: line 2: /nix/store/5zcrj0jp36ckjbxw5j4nj6b18nzid5rv-deploy-rs-0.1.0/bin/activate: Success
Connection to 192.168.1.251 closed.

Here is the corresponding deploy-rs node configuration:

nix
nixpi = {
 hostname = "192.168.1.251";
 sshUser = "pi";
 sshOpts = ["-t"];
 magicRollback = false; # In order for sshOpts "-t" to work, see URL
 profiles.system = {
 user = "root";
 # path = deploy-rs.lib.aarch64-linux.activate.nixos self.nixosConfigurations.nixpi;
 path = deploy-rs.lib.x86_64-linux.activate.nixos self.nixosConfigurations.nixpi;
 };
};

As you might notice I'm using "deploy-rs.lib.**x86_64-linux**.activate.nixos self.nixosConfigurations.nixpi". From what I understand while all my configuration has been cross-compiled, the final binary to run `activate` has been compiled native to my builder node so x86_64 instead of aarch64.

When I try to use "deploy-rs.lib.**aarch64-linux**.activate.nixos self.nixosConfigurations.nixpi" I get this error:

error: a 'aarch64-linux' with features {} is required to build '/nix/store/c6mljancr1wmnc0vqgwwwi8ic86krix4-builder.pl.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}
🚀 ❌ [deploy] [ERROR] Failed to push profile: Nix build command resulted in a bad exit code: Some(1)

While I could activate binfmt emulation using `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` and make this work the build time are atrociously slow to the point where a small config from scratch is taking almost 24h to build.

Is there a way to tell deploy-rs to use `pkgsCross.aarch64-multiplatform` instead?

As a temporary workaround, you can try to use `deploy-rs.lib.aarch64-linux.activate.nixos` with `remoteBuild = true` in order to build the entire profile on the target machine

Thanks for tip. This will work on some of my nodes, but some of them won't as they have less than 1G ram and just dies trying to build something

did you actually try this out? I tried something similar before and I don't remember it taking that long to build. what processor are you running this on?

Yes I tried. I'm running a i7-1185G7 but it's clocking a bit low (1.2Ghz, I need to take some time to configure throttled). When I have the bare minimum it's okish, around 8h but when I tried to add more stuff (an editor, nginx, ...) it is not manageable, build time explode.

For now I don't have a choice so that's what I'm doing, not updating often and not clearing my nix store. But it is not a solution in my opinion

do you what it's building? it should pull most stuff from the binary cache

@Skallwar

I may be mixed up or there's some issues in the way of it, but I am pretty sure this is not the case. The `activate` functions create an activatable derivation, an activatable derivation is the same as the original, but with the activation script, and `activate` binary included. The `activate` binary is executed on the remote host and manages the activation (stuff like rolling back when unconfirmed), so will need to be native to the target, and doesn't get used at all locally.

Sadly this doesn't work for me - I tried `remoteBuild = true;` and cli flag, but still get:

$ deploy --remote-build -d .#raspi-workshop
🚀 ❓ [deploy] [DEBUG] Checking for flake support
🚀 ℹ [deploy] [INFO] Running checks for flake in .
error: a 'aarch64-linux' with features {} is required to build '/nix/store/c6mljancr1wmnc0vqgwwwi8ic86krix4-builder.pl.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, ca-derivations, kvm, nixos-test, uid-range}
🚀 ❌ [deploy] [ERROR] Failed to check deployment: Nix checking command resulted in a bad exit code: Some(1)

And I don't know how to use binfmt idea as I'm not on NixOS... :cry:

In case anyone finds a solution, please let me know (I will too :yum: )

@notgne2

But the activate binary was built on the host pc not the remote one right? So if my host pc is x86_64 and the remote one is aarch64, the activate binary might be a x86_64 binary trying to run on an aarch64 machine?

`deploy-rs.lib.x86_64-linux.activate.nixos [derivation]` will give you an x86_64 activate binary, `deploy-rs.lib.aarch64-linux.activate.nixos [derivation]` will give you an aarch64 activate binary. There may or may not be errors attempting cross compiling it if you do it the latter way, but activation will fail if you don't.

I was using this one `deploy-rs.lib.x86_64-linux.activate.nixos` and it failed during activation.

Anyway, at stated in a previous comment, the error was on my configuration which forced cross compilation. Using `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` without the 2 lines mentioned work correctly and is fast enough using the cache.

Thanks for your help everyone

@tennox

Your distribution should have a package for this. For example on ubuntu: URL Installing this should fix your issue I think

Is there a reason `remoteBuild = true` doesn't solve this issue? I understand in some cases you want to build on the deploying machine with qemu, but as @tennox suggests, using remote build still throws an error complaining about the capabilities of the deploying machine.

The activation script still needs to be built on the deploying machine, as far as I understand.


Question: Vite CJS build warning caused by import of `unoCSSConfig` from `@tutorialkit/astro`?

@Barbapapazes Thanks for the pointer! This `mergeConfigs` function from `unocss` seems to be what we're looking for 👀

Vite CJS build warning caused by import of `unoCSSConfig` from `@tutorialkit/astro`
### Describe the bug

When TutorialKit is started in development or build mode, Vite's CJS build entrypoint error is shown:

$ pnpm dev
...
The CJS build of Vite's Node API is deprecated. See URL for more details.

### Steps to reproduce

sh
$ pnpm create tutorial

$ pnpm dev
...
The CJS build of Vite's Node API is deprecated. See URL for more details.

diff
$ code uno.config.ts

- import { unoCSSConfig } from '@tutorialkit/astro';

export default defineConfig({
- ...unoCSSConfig,

sh
$ pnpm dev

# No warnings

### Expected behavior

No warning should be shown

### Platform

- TutorialKit version: 0.1.4

Not sure why but seems to be caused by spreading `...unoCSSConfig` instead of using the variables from `import { rules, shortcuts, theme } from "@tutorialkit/theme";`

The UnoCSS config could also be made into a preset if that's easier:

`presetTutorialKit.ts`

ts
import { definePreset } from "unocss"
import { rules, shortcuts, theme } from "@tutorialkit/theme";

export default definePreset(() => {
 return {
 name: 'presetTutorialKit',
 rules,
 shortcuts,
 theme
 }
})

`uno.config.ts` (removing `...unoCSSConfig`)

ts
 presets: [
 presetUno({
 dark: {
 dark: '[data-theme="dark"]',
 },
 }),
 presetTutorialKit(),

I think this is happening when `unocss` imports the `uno.config.ts`, and that file imports `'@tutorialkit/astro'` that has imports for Astro stuff. Not sure why `unocss` decides to use CJS here.

We are planning to move most, if not all (?) `uno.config.ts` inside `@tutorialkit/*` packages so that end-users wouldn't have to maintain that file. Ideally tutorials wouldn't even have to create `uno.config.ts` unless they explicitly wanted to extend that. There's some discussion on URL

Roadmap to 1.0
Hey there!

Thanks for showing interest in using TutorialKit! We are currently planning to do a 1.0 release and we're excited to share a list of the changes that we are intending to make before the release.

#### Planned Changes

* [x] :rotating_light: Renaming `tutorialkit` to `@tutorialkit/cli`. So we can use that name for the vscode extension and also because the `npm create tutorial` command is the recommended way to create a new tutorial.
 * URL
* [x] :rotating_light: Renaming `@tutorialkit/components-react` to `@tutorialkit/react` and move it from `components/react` to `react`. Simpler name and descriptive enough.
 * URL
* [x] :rotating_light: Update `src/content/config.ts` to import the full schema from `@tutorialkit/types`.
 * URL
* [x] :rotating_light: UnoCSS config:
 * [x] Use a deep merge strategy and move default presets and transformers in `@tutorialkit/theme`.
 * URL
 * [x] Find a way to remove `content.inline`
 * URL
* [x] :book: Document docs on how tow write your own component with `tutorialkit:store`
 * URL
* [x] :book: Document that `tutorialkit:core` provide low level access to webcontainer and is to be used only as a last resort
 * URL
* [x] :book: Document that using `@tutorialkit/react` without TutorialKit is experimental
 * URL
* [x] Have `create-tutorial` depend on `"@tutorialkit/cli": "latest"` and release a fire and forget version.
 * URL
* [x] Remove `.vscode/settings.json`.
 * URL
* [x] Add TutorialKit extension to the list of recommended extension in the generated template.
 * URL
* [x] Extension:
 * [x] Frontmatter autocomplete:
 * #143
 * [x] Support folder-based and config-based ordering
 * #223
 * [x] Part, chapter, lesson deletion
 * #223

#### Nice to have, some post 1.0.0 launch:

* [ ] Have the default tutorial generated showcase all of the features of TutorialKit (ala slidev)
* [x] JSDocs on `TutorialStore` class
* [ ] VSCode Extension
 * [ ] Automated releases
 * [ ] Inheritance information
 * [ ] Drag and drop ordering for lesson / chapter / parts
* [ ] "Checks" to prevent navigation to next lesson until current checks have passed
* [ ] Syntax highlighting for
 * [x] Vue
 * #256
 * [x] Svelte
 * #212
 * [ ] Astro
* [ ] Monaco as an alternative option to CodeMirror
* [ ] Opt-in TypeScript intellisense

For the UnoCSS part, a similar solution to this one, URL used in Slidev could be used? This could help to simplify the DX and the starter.


Question: Update GOVERNANCE to include sub project voting?

Something else that I think would help with this discussion is: What do we need beyond just having a maintainer create the repo? What expectations or obligations are there that we need to satisfy for any new repo (whether freshly created due to contributions within the Cilium organization, or as a fork from an external repo for Cilium organization usage).

Update GOVERNANCE to include sub project voting
Outside of it being easier to find the governance of the project (e.g., I usually look for a GOVERNANCE.md file or a community repo), I'd like to ensure the voting is done in a transparent way.

As I understand it, all the governance is currently here: URL

I see that it is being followed by adding maintainers, e.g., URL

There's been confusion in public how certain decisions have been made in the project which is an opportunity for clarity and growth imho: URL

For tetragon, do you have an example of where the vote happened for this? I recall that when the project was being moved to CNCF, some of the votes were happening in the Cilium Slack which isn't a best practice and we recommended recording everything on github and/or mailing list. If a vote can't be dug up, the CNCF requests that you correct things to reflect what is stated in your governance. If you need help on how to do this, look at other CNCF projects e.g., URL

If you want to use a tool that helps, we have URL in CNCF that can help with voting.

Thanks for opening this issue, we are always trying to improve our governance process. We've gone through two governance reviews from TAG Contributor Strategy. See the latest one here. During the first one last January, it was pointed out that having votes in slack was not a best practice because they are quickly lost to the sands of capitalism. Since then, we have started to record the votes in the git history so we have a record of them. Unfortunately, the Tetragon vote was done before we made this switch.

I'm going to be working with an LFX mentee in the next quarter to update and improve the governance process for Cilium URL This will also be related to the work we will be doing for repo lifecycle URL I'll add this issue as a part of what we will be working on. Once we have this in place, I'll work with the committers to make sure we have a recorded history of the votes and continue to do so going forward.

Envoy seems to have a pretty simple process for this here URL

Repo lifecycle
As the project continues to grow and expand more repos are coming under the organization. It would be great to more formally define the lifecycle of repos under the project. We could take the work from Falco as an example. We should also consider how we handle forked repos.

Falco's repo lifecycle doesn't mention forks and it seems it's more targeted for "big projects" something that we would be expecting to see for things like hubble, tetragon, cilium-cli. However, we might need to define a process for smaller projects similar to URL URL and URL for example.

We should also add how handle experimental and unmaintained/archived repos.

Hi @xmulligan
I liked the lifecycle structure of falcosecurity

We can add more information about the repositories for better understanding of contributors

Handling folks will depend on the purpose of the folk , some folks are intended for personal exploration, contributing back to the original repository and alternative development or distribution. If the fork aims to become a separate project with its own development and maintenance, the developer should treat it as a distinct entity from the original cilium repository. If the fork is simply for personal use or experimentation, there's no need for any specific governance.

We can only handle forks whose intentions are to contribute changes back to the original repository. It would be nice to place rules that fight against spam in pull requests and misleading forks. Please , is there any thing else we should consider with forks ?

Hello Everyone!
I would like to work on this issue as part of the LFX Mentorship.

Hi @xmulligan! I'm Abhay. First and foremost, the project is thriving. I have successfully applied to the mentorship program and completed all the pre-tasks. I am eagerly awaiting the opportunity to work with you, learn more about the projects, and delve into interesting challenges.

Hi there! I am Rudraksh, the project on Cilium governance documentation looks pretty interesting to me. While considering the points mentioned by @xmulligan @joestringer & @aanm. I performed a data analysis on the repos under Cilium project using Pandas. I came up with some figures with Cilium as:

image

I am looking forward to leveraging it further, to get more details about the individual projects which can help while defining lifecycle. While I found the Falco project repo lifecycle structure to be an excellent reference, I understand that we may need to make certain adjustments in the context of Cilium. I also came across CNCF governance templates which has been incredibly helpful in learning about governance in open source.

In the latest, review of Cilium governance the essence of contributor ladder was one of the major feedback/reviews by TOC, is this an aspect we should prioritize in the context of the Cilium governance project?


Question: provide one method or api so that the upper layer can know the backup progress?

The issue is that rdiff-backup expects the target directory to it alone. A system directory like this `.WD...` directory can only create issues. Either remove it or backup to some sub- directory of W:.

At this stage, I don't see any other short term solution. Longer term would be the implementation of #790 or a new enhancement to simply ignore certain files.

provide one method or api so that the upper layer can know the backup progress
## Is your feature request related to a problem? Please describe.

Currently myapp.exe will call rdiff-backup.exe, but did not know the progress on the backup

## Describe the solution you'd like
simple solution is: rdiff-backup.exe write the progress info to one log file
myapp.exe can read the real time log from that file.

## Describe alternatives you've considered

## Additional context

I'm not sure what you expect. rdiff-backup doesn't know of a progress in terms of percentage or time. And each processed file is already listed, starting with verbosity 5. And there is already a logfile with this information.

1. like a big file copying, can we know the percentage?
2. where is the current log saved?

what I want is to show the progress when starting backup.

the call flow is: myapp.exe launch rdiff-backup.exe as I posted on another issue.
 URL

Thanks. And I think **backup.log** is good enough for my case.
One more thing is this log file will be rotation or not?
Currently as I checked, it has 20M+ size.

backup failed by "PermissionError: [WinError 5] Access is denied: b'W:/.WDSyncHistory/.WD Hidden Files'"
## Bug summary

rdiff-backup.exe --force -v5 --print-statistics D:/imagebk W:/
failed by access denied on hidden files.

## Version, Python, Operating System

yaml
exec:
 api_version:
 actual: 0
 default: 200
 max: 201
 min: 200
 argv:
 - *****/rdiff-backup/rdiff-backup.exe
 - info
 parsed: null
 version: 2.2.3
python:
 executable: *****\rdiff-backup\rdiff-backup.exe
 name: cpython
 version: 3.10.7
system:
 fs_encoding: utf-8
 platform: Windows-10-10.0.22000-SP0

## rdiff-backup call

rdiff-backup.exe --force -v5 --print-statistics D:/imagebk W:/

## What happened and what did you expect?

copy the folder from D:/imagebk to W:/

## More information

WARNING: this command line interface is deprecated and will disappear, start using the new one as described with '--new --help'.
* Using repository 'W:/'
WARNING: Target path 'W:/' does not look like a rdiff-backup repository but will be force overwritten
* Hardlinks disabled by default on Windows
* Unable to import module (py)xattr. Extended attributes not supported on filesystem at path D:/imagebk
* Unable to import module posix1e from pylibacl package. POSIX ACLs not supported on filesystem at path D:/imagebk
* -----------------------------------------------------------------
Detected abilities for source (read only) file system:
 Access control lists Off
 Extended attributes Off
 Windows access control lists On
 Case sensitivity Off
 Escape DOS devices On
 Escape trailing spaces On
 Mac OS X style resource forks Off
 Mac OS X Finder information Off
-----------------------------------------------------------------
* Directories on file system at path W:/rdiff-backup-data/rdiff-backup.tmp.0 are not fsyncable. Assuming it's unnecessary.
* Unable to import module (py)xattr. Extended attributes not supported on filesystem at path W:/rdiff-backup-data/rdiff-backup.tmp.0
* Unable to import module posix1e from pylibacl package. POSIX ACLs not supported on filesystem at path W:/rdiff-backup-data/rdiff-backup.tmp.0
* -----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
 Ownership changing Off
 Hard linking On
 fsync() directories Off
 Directory inc permissions Off
 High-bit permissions On
 Symlink permissions Off
 Extended filenames On
 Windows reserved filenames On
 Access control lists Off
 Extended attributes Off
 Windows access control lists On
 Case sensitivity Off
 Escape DOS devices On
 Escape trailing spaces On
 Mac OS X style resource forks Off
 Mac OS X Finder information Off
-----------------------------------------------------------------
* Backup: escape_dos_devices = False
* Backup: escape_trailing_spaces = False
* Enabled use_compatible_timestamps
NOTE: Symbolic links excluded by default on Windows
* Given repository doesn't need to be regressed
NOTE: Starting mirror from source path D:/imagebk to destination path W:/
* Processing file .
* Processing file .WDSyncHistory
* Processing file .WDSyncHistory/.WD Hidden Files
* Processing file Garbage Dispenser2.mp4
* Cleaning up
Traceback (most recent call last):
 File "rdiffbackup\run.py", line 170, in
 File "rdiffbackup\run.py", line 37, in main
 File "rdiffbackup\run.py", line 105, in main_run
 File "rdiffbackup\actions\backup.py", line 159, in run
 File "rdiff_backup\backup.py", line 39, in mirror_compat200
 File "rdiff_backup\backup.py", line 197, in patch
 File "rdiff_backup\rorpiter.py", line 142, in __call__
 File "rdiff_backup\rorpiter.py", line 179, in _finish_branches
 File "rdiff_backup\backup.py", line 641, in end_process_directory
 File "rdiff_backup\rpath.py", line 807, in rmdir
PermissionError: [WinError 5] Access is denied: b'W:/.WDSyncHistory/.WD Hidden Files'
[16948] Failed to execute script 'run' due to unhandled exception!

Were the three files listed before the "cleaning up" message existing on W: before you started the first backup? What does `dir /s w:` show? You're aware that you're going to lose everything previously present on this drive?

what do you mean?
The 2 of 3 files are not actually files. They are dot and one folder begin with dot

Definitely, I want to get help.
 The W:\ disk is map from network disk. And I do did some backup before. But now I just want to copy from **D:/imagebk to W:/**, and meet the issue as I posted.

image

There is one hidden folder:

image

Could you help to check? thanks.