Question: Introduce flow control between partitions?

We're working on another solution for that in parallel. URL

I know the issue specifies IPC, but the proposed solution will enforce rate limits between the different categories, user submitted commands, internal/scheduled commands, and remote ones.

Introduce flow control between partitions
**Description**

Partitions send commands between each for the following cases:

- Message correlation (including correlation)
- Deployment distribution

In both cases, since communication is assumed to be unreliable, we have built-in retry mechanisms. In the event of a failure in these, or a bug on either side, it can happen that the sender overloads the receiver. Or it can be that the receiver is generally overloaded and the sender shouldn't send anything until the situation resolves itself.

There's no way for the receiver to notify the sender at the moment that it should slow down. We saw this in one incident, where the deployment distributor kept retrying deployments at a faster rate than the receiving partition could process, resulting in the build up of an incredibly large log and that partition being effectively unavailable from a user point of view.

Ideally, we should see a solution which satisfies the following:

- [ ] Commands from other partitions can be rejected based on the current load, sharing the same rate limiter as external commands
- [ ] Certain commands may need to be whitelisted (e.g. backup/checkpoint related commands)
- [ ] The receiving partition can provide information about when to retry
- [ ] The sending partition does not buffer commands for retry in an unbounded way

The last point is to ensure we don't run out of memory by buffering commands to be sent to a given partition which is unable to process any commands for a longer period of time.

Probably a M size issue (up to 6 weeks), and it is not urgent if not something else goes wrong. We should look at it as an improvement for next quarter, and evaluate if there is any overlap with other initiatives around inter-partitions communication, i.e. switching to gRPC.

affected partition stability in an incident URL
2 partitions got overloaded with message subscription create commands and the backlog was growing at a higher rate than the processing

Planning:

- Time box 8 hours to propose a solution/breakdown for this

@oleschoenburg and @deepthidevaki will look into it in the next two weeks

@deepthidevaki and I looked into this a bit from the angle of "let's improve flow control in general". In particular, URL was at the back of our minds.

We think that a first step in the right direction would be to remove the `LogStorageAppender` in general and replace it with a simple limiter that sits in front of the `Sequencer` to protect the raft thread from overloading it with too many writes. We think that a fixed limit or a simple RTT based limit would work well for this.

As a second step, we can move the flow control for user commands away from the `CommandApi` and implement it in front of the `Sequencer` instead. The idea here is to have one common place where flow control for commands is implemented. User commands would then acquire permits from two limiters, the configured one for user commands (AIMD by default), and the raft thread limiter.
To implement this, I propose a new class that sits in front of the `Sequencer`. The `Mixer` would decide which flow control to use, based on the _source_ of records. Sources would be _User Commands_, _Processing Results_, _Internal Commands_ and _Remote Commands_.

**After implementing the first two steps, the system behavior should remain almost the same**. User commands go through a configurable limit while other records such as processing results, internal commands (e.g. timer triggers), and remote commands (e.g. message correlation) are only held back by the limiter that protects the raft thread.

Building on top of this, we can think about adding flow control for _Remote Commands_. The challenge here is that we probably can't just drop commands silently. Let's take message correlation as an example: When a _Correlate_ command is sent to a remote partition, the sent time is updated immediately and the correlation is only attempted again after 10 seconds. If we were to introduce a limiter for remote commands, we might run into problems because every dropped command will result in at least 10 seconds of delay. Ultimately, a solution similar to URL would be desirable. The difference here is that we must wait for the command to be received, written _and committed_ on the remote partition. Currently, this is not possible because inter-partition communication is a one-way channel from sender to receiver.
Instead, we can extend the inter-partition command sender to take a completion callback that is executed whenever the command is written on the remote partition. We can then use that for message correlation (and possibly other performance sensitive features such as signals) to shorten the retry interval for messages that were not written on the receiver.

[tasklist]
## Refactoring flow control
- [ ] Let the `Sequencer` write directly to the raft thread
- [ ] Implement a `Mixer` component in front of the `Sequencer`
- [ ] The `Mixer` protects the raft thread from overloading
- [ ] The `Mixer` provides metrics for flow control state
- [ ] Record writers are categorized by use-case
- [ ] The `Mixer` applies flow control for user commands
- [ ] Cleanup: Building a `LogStreamWriter` should not be async

A prequel to Ole's comment above.

We prototyped flow control in Sequencer. We added separate request limiters for user commands, remote commands and internal commands. Processing result from stream processor is never rejected. All limiters are configured to use AIMD. Limiters for internal and remote commands are configured to be more lenient to ensure internal backlog can make progress even if there is high load from users, but this is up for discussion.

Although our main goal is to limit remote commands, we first evaluated with a high load of timers because it was the easiest to setup a benchmark with timers. **Result: The cluster with the limiters for internal-commands is still running strong after a couple of days. The reference benchmark is almost dead now with 400Mil of processing queue size.**

Reference
image

POC with flow control for user, remote and internal commands.
image

Next question is how this type of flow control works for remote commands. As Ole explained before, rejecting remote commands can cause higher delay in message correlation mainly because we are not retrying aggressively enough. In a very high load, similar to what we saw in the support incident, it is probably fine. But in a normal load scenario we have to ensure that the messages are still correlated in real-time. We have not evaluated this yet. So as a next step, I would like to evaluate the impact of flow control for remote commands. Based on the results, we can decide whether we want to add request limiters for remote commands in the Sequencer (together with better retry strategy on the sender) or add caching similar to the timers proposed for #13870 or a combination of both.

Nevertheless, we should take the next steps proposed by Ole above.

@deepthidevaki just want to mention that we already implemented caching for the writes in URL

I think caching for remote commands is gonna be really tricky. I suppose you could do it on the receiver side :thinking: That would reduce a bit the load on the remote commands for back pressure I guess.

True. We don't have a prototype for it yet.

We discussed the following for backpressure approach
1. Buffering in the receiver and receiver can retry (limit the number of retries) if the write was rejected due to backpressure. This can put more pressure on receiver which might be already overloaded.
2. Buffering in the Inter-partition sender and retry if the write was rejected due to backpressure. (limit the number of retries). The record is already buffered in the engine, but we have to buffer it again in inter-partition sender.
3. Somehow the recurrent checkers in engine should know to retry faster if it was rejected due to backpressure. Needs more information exchange between the inter-partition transport and the checkers.

1 and 2 is in the direction of providing more reliable transport for inter-partition.

Alternate is to follow the caching approach used for timers, to reduce the chance of backpressure due to sender repeatedly trying to send the same commands. But here the most tricky is that the we can mark it as written only when the receiver has written and committed the command. Also how to handle if the command was rejected -> should retry again?

Adding backpressure will be the easiest part to implement, but the problem is that in specific workloads there is a chance that a few messages get delayed forever because they were unlucky and get caught in the backpressure loop.

It was unclear to me before whether we want to add flow control for internal commands now that we have the caching solution. It seems caching did not fully solve the problem encountered by a user. Flow control for internal commands would likely solve the reported issue.

So I'm putting this issue back to inbox to discuss it in next traige/planning. It would make sense to start implementing the changes proposed in the above comment even if we don't have a completed solution for IPC.

@deepthidevaki can you link the prototype PR/branch here? thanks!

Hi.

I've tested this case URL Seems like queue size is really capped now (was up to 1mil events, now up to 500k events). But this the only good news. Overall performance became lower, windows of unavailability became wider. It's reasonable, because number of timer events doesn't change it's just processed slower now.

image

I've expected that with limited processing of timer events - there will be some "bandwidth" for processing new events (create new PIs). That' doesn't happen.

This could be because in the poc, we deliberately configured flow control for user commands and internal commands differently. We are more aggressively rejecting user commands while accepting more internal commands.

Scheduled tasks should avoid overloading the streamprocessor
**Description**

The workflow engine employs the help of several scheduled tasks that periodically append commands to the log stream. This helps offload time-consuming work that could otherwise block the actual workflow processing.

For example, job deadlines are checked for expiration by the `JobTimeoutTrigger`, which will append commands for each job to time them out. The stream processor will then process this command asynchronously. Eventually, the job changes state to `TIMED_OUT` when the command is processed.

In the meantime, the `JobTimeoutTrigger` continues to check for expired job deadlines periodically. If the stream processor is blocked, it may happen that the `JobTimeoutTrigger` continues to append the same `Job:TIME_OUT` command. There is nothing that stops it from overloading the stream processor.

In fact, many of the scheduled tasks suffer from a similar problem.

**Proposed solution**

Scheduled tasks need to keep a cache of recently appended commands (or something similar) that can be used to avoid appending the same command multiple times.

This should be available in a generic way, so all scheduled tasks can easily add support for this.

**Incident**

Follow up issue from incident: INC-274: Streamprocessor lagging behind

Support:
- URL
- URL

Marking this as high impact, as it is a recurring pain. Resolving this would lower the load on clusters experiencing problems where timers, buffered messages, or job timeouts play a role. This would help in the investigation as well as ease recovery.

I'm increasing the priority of this issue/feature to 'upcoming' since there is a support case waiting for it.
Still, it is highly unlikely to be able to work on this before 8.4. However, we will update the issue once we plan to work on it.

ZDP-Planning:
- pushing into the inbox of ZPA to get an update on this as it relates to #14003

The ZPA-team discussed this issue again and now considers this a bug. Originally, I felt it was a feature, as we have never supported many timers triggering at once (i.e., there's always a limit somewhere). However, the team feels that outages should not occur from existing features, irrespective of how they are used. Add to this that we don't have documentation about some artibrary limit to the number of timers. Lastly, users feel that this is a bug, and so should we.

As a bug, it may be easier to prioritize and release a fix for this as we can backport and patch previous versions.

@korthout Hi!

Is there any updates for this bug? I would be nice to have a due date )

Hi @dddpaul, the team plans to spend time on this in our current iteration. There's a chance that we can fit a solution in the upcoming patch releases, but we cannot promise this.

We'll need to consider different solutions. For example, the proposed cache reduces the load on the system in most cases but will not remove the problem completely. Alternatively, ideas like #12560 could help but come with other downsides. The developer(s) working on this issue should consider whether caching is enough or whether another solution might be more fitting. As this is not yet clear, it's possible that a solution will not be ready before the upcoming patch releases. However, we plan to spend time on this issue now.

There was an L1 blocker again because of this URL

Handled by @npepinpe

Users are heavily impacted by this, so I would like to bump this to critical, with a focus on fixing timers. Timers are different from message correlation has they don't involve inter-partition communication, which may help narrow the scope of the fix for now (i.e. writes are guaranteed to succeed).

That said, I noticed some possible concurrency issues with the `DueDateTimerChecker`, which could lead to more timers than required being scheduled under high load.

If enabled, the `DueDateTimerChecker` runs in the asynchronous scheduler actor, separately from the stream processor.

However, both actors will access the `DueDateTimerChecker`:

`DueDateTimerChecker#scheduleTimer` is called from the engine/side-effects, i.e. processing actor. It then:

- Accesses the following state: `shouldRescheduleChecker`, `checkerRunning`, `nextDueDate` (rw)
- Checks if the checker is currently running. This is a non-volatile boolean check.
- If it's not running, it schedules a task.

At the same time, scheduling is done on the async scheduler actor. This is the `TriggerEntitiesTask`. It:

- Accesses the following state: `shouldRescheduleChecker`, `checkerRunning` (rw), `nextDueDate` (rw)

Is this correct? If so we have non-volatile state being accessed by two different actors, and possibly some race conditions on when and if to schedule the next timer.

Moving on the the possible solution. In order to keep it light of course, we'd need to to have a way to identify a timer trigger uniquely. Since every time instance can only be scheduled for trigger once in the state, it should be enough to distinguish them with the timer key, which is a single long.

The general idea, then, is to cache locally in the `DueDateTimerChecker` state (or some state) which commands have been written, so as to avoid writing them again. A perfect cache would be unbounded - you'd have to possibly store as many timers in there as could possibly exist (worst case scenario). But a lossy cache is probably acceptable. Take `LongHashSet` from Agrona - an empty, pre-initialized set instance of 1 million keys takes 8MB of memory. Filled, it takes about twice, ~16MB. It's a bit hard to add LRU behavior to such a set however. Eviction could be random to keep things simple/fast, or we can use a `SortedSet`. However, a `TreeSet` takes about ~60MB for 1 million longs, so it's already substantially heavier - but it's pretty easy/fast to add LRU eviction to it. So on the off chance the key was evicted _before_ it was removed explicitly from the cache, you'd potentially still write a second trigger command. But the bigger the cache, the less likely this is to happen.

I sync'd with @Zelldon today about this, and we decided to move forward with the local cache, only for checkers which perform no inter-partition communication.

For timers, we'll use the timer key as the unique cache ID. When iterating of timers, if the timer key is in the cache, then we skip it. Once the task result is built, we can populate the cache with all the timer keys we will be writing. Since the scheduler (even async) is sequenced, we're guaranteed that this write will succeed, or the whole thing will fail (and the cache would be "restarted"). We won't be touching things like leader election and so on. On recovery, it's still possible that the same timers would be triggered twice, but the overload load will be greatly reduced.

This solution is likely adaptable to other checkers which run locally, such as job timeout and message expiry. For messages we can easily use the key. For jobs, we can't use the job key, since you can have multiple activations/time outs. We could use the job key if we can guarantee that the cache is properly cleaned up whenever the job is activated.

Anyway, long winded comment to say we'll get started on a bounded lossy cache-based prototype.

So in the end, we'll build the cache on the stream processor side. I describe the approach we ended up with in the description of this PR: URL

Let me know if you have questions. For now it's entirely on the stream platform/ZDP side, but it's still useful to know how it works. Plus, if you want to cache future scheduled commands in the future, you'll need to know where to add them :upside_down_face:

See some benchmark results courtesy of @Zelldon - URL :tada:

We took it off the ZPA board, as ZDP has found a solution on their side 🙇 Thanks again @Zelldon and @npepinpe

Hi!

This looks like a wonderful job. But this new timer cache doesn't solve my problem with bunch of UNIQUE timers from DIFFERENT process instances. I've illustrated my problem here URL My comment was stated as duplicate of this issue which was likely fixed with timer cache solution in PR #15136 (for 8.3).

I could not understand how this cache solution will help me but had some hope and belief. Well, there no miracle happened.
I've just tested on 8.3-stable
image

Thousands or millions of unique timer events still overload brokers:
image

This is my process with timer:
image


Question: Graceful shutdown consistently takes 90 seconds?

Should be fixed now on main branch, please update this flake and remove any workarounds you might have had. Feel free to reopen if you're still having issues!

Graceful shutdown consistently takes 90 seconds
Jan 13 02:50:04 azalea systemd[1]: Reached target System Power Off.
░░ Subject: A start job for unit poweroff.target has finished successfully
░░ Defined-By: systemd
░░ Support: URL
░░
░░ A start job for unit poweroff.target has finished successfully.
░░
░░ The job identifier is 6768.
Jan 13 02:50:04 azalea systemd[1]: Shutting down.
Jan 13 02:50:04 azalea systemd-shutdown[1]: Syncing filesystems and block devices.
Jan 13 02:50:04 azalea systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Jan 13 02:50:04 azalea systemd-journald[662]: Received SIGTERM from PID 1 (systemd-shutdow).
Jan 13 02:50:04 azalea systemd-journald[662]: Journal stopped
░░ Subject: The journal has been stopped
░░ Defined-By: systemd
░░ Support: URL
░░
░░ The system journal process has shut down and closed all currently
░░ active journal files.
░░ The unit UNIT completed and consumed the indicated resources.
Jan 13 02:48:47 azalea kernel: ucsi_acpi USBC000:00: unknown error 256
Jan 13 02:48:47 azalea kernel: ucsi_acpi USBC000:00: GET_CABLE_PROPERTY failed (-5)
Jan 13 02:50:03 azalea systemd[1]: session-3.scope: Stopping timed out. Killing.
Jan 13 02:50:03 azalea systemd[1]: session-3.scope: Killing process 1879 (cosmic-settings) with signal SIGKILL.
Jan 13 02:50:03 azalea systemd[1]: session-3.scope: Killing process 1882 (ctrl-c) with signal SIGKILL.
Jan 13 02:50:03 azalea systemd[1]: session-3.scope: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: URL
░░
░░ The unit session-3.scope has entered the 'failed' state with result 'timeout'.
Jan 13 02:50:03 azalea systemd[1]: Stopped Session 3 of User sebastian.
░░ Subject: A stop job for unit session-3.scope has finished
░░ Defined-By: systemd
░░ Support: URL
░░
░░ A stop job for unit session-3.scope has finished.
░░
░░ The job identifier is 6778 and the job result is done.

I'm not entirely sure if this is something related to my hardware or with this flake. Has anybody else been able to reproduce?

The `ucsi_acpi` stuff is a Framework laptop red herring.

Jan 13 03:10:14 azalea cosmic-session[1831]: EXITING: received request to terminate
Jan 13 03:10:14 azalea cosmic-session[1831]: process 'ProcessKey(2v1)' cancelled
Jan 13 03:10:14 azalea systemd[1]: Stopping Session 3 of User sebastian...
░░ Subject: A stop job for unit session-3.scope has begun execution
░░ Defined-By: systemd
░░ Support: URL
░░
░░ A stop job for unit session-3.scope has begun execution.
░░
░░ The job identifier is 2118.
Jan 13 03:11:44 azalea systemd[1]: session-3.scope: Stopping timed out. Killing.
Jan 13 03:11:44 azalea systemd[1]: session-3.scope: Killing process 1893 (cosmic-settings) with signal SIGKILL.
Jan 13 03:11:44 azalea systemd[1]: session-3.scope: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: URL
░░
░░ The unit session-3.scope has entered the 'failed' state with result 'timeout'.
Jan 13 03:11:44 azalea systemd[1]: Stopped Session 3 of User sebastian.
░░ Subject: A stop job for unit session-3.scope has finished
░░ Defined-By: systemd
░░ Support: URL
░░
░░ A stop job for unit session-3.scope has finished.
░░
░░ The job identifier is 2118 and the job result is done.
Jan 13 03:11:44 azalea systemd[1]: session-3.scope: Consumed 57.600s CPU time, 1.6G memory peak, 364M read from disk, 348K written to disk.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: URL
░░
░░ The unit session-3.scope completed and consumed the indicated resources.

It looks like for some reason `cosmic-settings` is holding up the graceful termination of the user session?

Some more seemingly relevant logs:

Jan 13 03:10:14 azalea cosmic-session[1831]: EXITING: received request to terminate
Jan 13 03:10:14 azalea cosmic-session[1831]: process 'ProcessKey(2v1)' cancelled
Jan 13 03:10:14 azalea pipewire[2149]: mod.x11-bell: X11 I/O error handler called on display :1
Jan 13 03:10:14 azalea pipewire[2149]: mod.x11-bell: X11 display (:1) has encountered a fatal I/O error
Jan 13 03:10:14 azalea .xdg-desktop-po[2488]: Error reading events from display: Broken pipe
Jan 13 03:10:14 azalea org.freedesktop.secrets[2545]: discover_other_daemon: 1
Jan 13 03:10:14 azalea systemd[1786]: xdg-desktop-portal-gtk.service: Main process exited, code=exited, status=1/FAILURE
Jan 13 03:10:14 azalea systemd[1786]: xdg-desktop-portal-gtk.service: Failed with result 'exit-code'.
Jan 13 03:10:14 azalea systemd[1786]: cosmic-app-library.scope: Consumed 171ms CPU time, 67M memory peak.
Jan 13 03:10:14 azalea systemd[1786]: cosmic-osd.scope: Consumed 177ms CPU time, 70.2M memory peak.
Jan 13 03:10:14 azalea systemd[1786]: cosmic-bg.scope: Consumed 298ms CPU time, 129.9M memory peak.
Jan 13 03:10:14 azalea systemd[1786]: xdg-desktop-portal-cosmic.scope: Consumed 221ms CPU time, 64M memory peak.
Jan 13 03:10:14 azalea systemd[1786]: cosmic-files-applet.scope: Consumed 540ms CPU time, 65.3M memory peak.
Jan 13 03:10:14 azalea pipewire[2149]: pw.node: (alsa_output.pci-0000_c1_00.6.analog-stereo-59) graph xrun not-triggered (0 suppressed)
Jan 13 03:10:14 azalea pipewire[2149]: pw.node: (alsa_output.pci-0000_c1_00.6.analog-stereo-59) xrun state:0x7ff18403b008 pending:1/1 s:934034164248 a:934034287623 f:9>
Jan 13 03:10:14 azalea systemd[1786]: app-cosmic-com.system76.CosmicAppList-2759.scope: Consumed 4min 29.541s CPU time, 4.1G memory peak.
Jan 13 03:11:44 azalea systemd[1786]: Activating special unit Exit the Session...
Jan 13 03:11:44 azalea systemd[1786]: Stopped target Main User Target.
Jan 13 03:11:44 azalea systemd[1786]: Stopped target Current graphical user session.
Jan 13 03:11:44 azalea systemd[1786]: Stopped target Startup of XDG autostart applications.
Jan 13 03:11:44 azalea systemd[1786]: Stopped target Cosmic Session Target.
Jan 13 03:11:44 azalea systemd[1786]: Stopped target Session services which should run early before the graphical session is brought up.

I'm not really sure why `Activating special unit Exit the Session...` is delayed by 90 seconds.

I think this `cosmic-session[1831]: process 'ProcessKey(2v1)' cancelled` is the key line here. Not sure what `ProcessKey(2v1)` means but that's the last line for `cosmic-session` in that boot. I think this line indicates that for some reason, `cosmic-session`'s graceful shutdown handling is getting interrupted by something so it can't tell systemd that it's done shutting down.

Just saw URL looks like i'm currently on an older commit than the fix for that so I'm gonna update and see if that fixes things.

├───nixos-cosmic: github:lilyinstarlight/nixos-cosmic/847b93e3b63bcea9a477dd86bb4b56ce7e051f0e?narHash=sha256-Tm%2BBsKXJS/EdJd9DvLxDbw%2BchPI1o7A9RHKIFxho36I%3D (2024-12-25 01:36:23)

Shutdown takes a long time if GeoClue2 service is not enabled
Not sure if this is a bug or not, but I noticed that shutdown my machine would take a long time (around 1 to 2 minutes) because I didn't have the `GeoClue2` service running and because of that I would get these error messages:

cosmic-session[1938]: Failed to watch theme org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.GeoClue2 was not provided by any .service files. Will try again in 25s

This would make systemd timeout trying to stop session 3 of my user. Taking around 1m 30s during the shutdown process.

Adding this to my config seems to fix the issue:

services.geoclue2.enable = true;

Maybe a note about this should be in the troubleshot part of the README?

Thank you for finding out the reason for session 3 interrupting shutdown!


Question: Confusing which Xcode I'm using with a nightly toolchain?

There's also the option of just adding to the terminal environment directly using the `ExtensionContext`:

typescript
context.environmentVariableCollection.clear();
context.environmentVariableCollection.prepend(
 "PATH",
 "/path/to/usr/bin:",
 { applyAtShellIntegration: true }
);
for (const variable in configuration.swiftEnvironmentVariables) {
 context.environmentVariableCollection.replace(
 variable,
 configuration.swiftEnvironmentVariables[variable],
 { applyAtShellIntegration: true }
 );
}

Using this approach the user will have the environment automatically populated in their VS Code terminal without having to launch a special profile. The only minor annoyance being that modifying certain variables (e.g. `PATH`) triggers a warning before the terminal will be updated:

<img width="522" alt="Screenshot 2024-06-27 at 2 30 02 PM" src=" URL

Confusing which Xcode I'm using with a nightly toolchain
Now that Xcode 16 Beta is out I have two Xcodes installed on my machine. I have also downloaded the nightly build of the 6.0 Swift toolchain. Sometimes I want to use it with Xcode 15, sometimes with Xcode 16 Beta. When I use the command to select a Swift toolchain it's confusing which Xcode it's actually using. It would be nice if I could select that separately from the toolchain selection. At the end of they day, they are different environment variables on Mac, DEVELOPER_DIR and TOOLCHAINS.

When you select an Xcode installation you are selecting the toolchain from that Xcode install. If you select a toolchain install it isn't using your Xcode install at all, outside of one thing, the xctest library. I'm not certain offhand which Xcode version of xctest it would be using. Maybe @matthewbastien who wrote the toolchain select can answer that.

If you are using swift-testing selecting a toolchain install should mean you aren't using an Xcode install at all. In theory you shouldn't even need Xcode installed. I haven't tested that though.

We are not using the DEVELOPER_DIR or TOOLCHAINS environment variables.

Just for clarity, swift-testing is included in the Xcode 16 Beta. This means that if you are using Xcode and then switch to a toolchain install you may need to explicitly specify swift-testing as a dependency.

You'd need to do the same if your package is compiled on Linux. I am assuming this will not be necessary once Swift 6 has a proper release though.

I wonder if that's what we should be doing on macOS. That would allow us to be consistent with the command line experience.

Though as it is, if you swift build from the terminal inside VSCode you actually get different settings right now.

Yeah not ideal

Even if we used DEVELOPER_DIR we'd have this situation. We cannot affect the environment variables for the whole of VSCode, only the processes we start eg swift build etc. The terminals are setup by VSCode not the extension. There is a thing called a terminal profile, which can be used to setup a custom environment See #914 , I don't know if that affects all terminals, or add a new type of terminal.

My thoughts so far are that there are two scenarios here when the user selects a toolchain:

1. An Xcode toolchain is selected in which case we should be setting `DEVELOPER_DIR` to the Xcode path.
2. A public/nightly toolchain is selected which is a bit trickier. If there are multiple Xcodes installed on the system then we should ask the user which Xcode they want to select - like the `Swift: Select Xcode Developer Dir...` command we used to have before the toolchain selection obsoleted it.

It really boils down to the fact that we should be setting `DEVELOPER_DIR` if there are multiple Xcodes installed on the system. The `TOOLCHAINS` environment variable is also straightforward to set alongside our `swift.path` configuration which I can experiment with. I'm working on a PR to push up that folks can try out.

A twist in all this is that the CommandLineTools doesn't support the `TOOLCHAINS` env var. That makes me much less enthusiastic about relying on it.

For cross platform, I can see a scenario where the user does an `xcode-select --install` to get a minimal development environment and then installs the latest Swift toolchain for things like Swift Embedded. But we'll need to actually set the PATH to the toolchain to pick that up properly.

Swift terminal profile
**Is your feature request related to a problem? Please describe.**
I've made the mistake a few times where I have select a toolchain in vscode and it has been built for via tasks. I then go to the integrated vscode terminal to issue to run some swift command that is a different version of swift.

**Describe the solution you'd like**
You can provide different terminal profiles URL so we could provide a Swift terminal profile with selected toolchain in the PATH so that user can select this swift profile in settings, for example `terminal.integrated.defaultProfile.osx`

**Describe alternatives you've considered**
N/A

**Additional context**
N/A

@matthewbastien so that appears every time you start up a terminal window or only for existing terminal windows?

@adam-fowler the warning will only appear for existing terminal windows that need to be reloaded to capture new environment variables.

@matthewbastien if I disable `swift.enableTerminalEnvironment` and then create a new terminal with the swift profile, `swift.swiftEnvironmentVariables` values are added but `which swift` still indicates `/usr/bin/swift` is being used not the selected toolchain path.


Question: Better ways to introspect? --> JSON representation of HDC-API?

Thanks to the improvements obtained in #21, we should certainly allow this.
The reason being, that custom feature states are much more valuable if their meaning is documented.
(Lesson learned from the "Architecting a Bridge" manifesto: Automate documentation!)

Note how there's no need to make this mandatory. Firmware developers who do not need this feature can just omit it.

Better ways to introspect? --> JSON representation of HDC-API
Currently it's a cumbersome mixture of very specific Commands and Properties, which don't scale nicely.
It also feels as if it weren't tailored to its use-cases.

Discarded as bad Idea: A single command with multiple arguments:
 - GetIntrospection(item_type=command, item_id=0xF0, aspect_id=0xF0)
 - That's cumbersome, too, because we need to introduce new tables for item_types and aspect_ids
 - Command returns var data-type

Potentially better idea: A single command without any arguments, which returns a machine-parseable syntax detailing all details about a whole feature. What could such a simple enough syntax be, which …
 - … a microcontroller can generate on the fly, by composing it directly into the TX buffer.
 - … a tool can translate into source-code for proxy-stubs or for descriptors for devices implemented in other languages, i.e. a "digital twin" implemented in Python.
 - … a host can use to dynamically create proxy objects.
 - … is human readable.

 How about JSON?
Find out by implementing a proof-of-concept firmware that generates it on-the-fly from descriptor data.

Related to #16 and #18, which might benefit from a more scalable solution that may allow to include or omit those aspects, without having to change the API of mandatory Commands/Properties.

To get a feeling about how it would look like, I quickly implemented a descriptor serializer that produces strict JSON in Python:
JSON_minimal_device.txt

It is surprisingly feasible and convenient to have a C implementation of the HDC packetizer, which is able to cope with a stream of data whose length is unknown ahead of time. This is a prerequisite for the device firmware to be able to reply with a dynamically generated JSON representation of its HDC interface.
Implemented it as HDC_Compose_Packets_From_Stream(), which in 812b171ac8158b3d6aeacde0243294a39d32e64c still coexists with its predecessor HDC_Compose_Packets_From_Pieces(), but will soon replace it.

Surprisingly the new implementation is not only more capable, but also has a smaller footprint:
image

Although re-implementing the C packetizer was initially motivated to enable the streaming of dynamically generated JSON, it turns out that it also had advantages for the existing use-cases.
The new implementation has a smaller footprint and its source-code is much more readable.
See commit: 47afdbb0eab29e84bc4a1c698975a1f2c8738cfa
image

Name and docstring for each State a Feature may be at.
Which are the states and their meanings that each featrure's state-machine implements?
Currently reference-implementations use an poor in between: A free-form string Feature.FeatureStatesDescription property with a recommendation on how the syntax could be. That's a recipe for disaster.
This is similar to #16 , the actual question is: How much introspection do we need? Is it sufficient to address it like a free-form docstring, or shoul we go for strict machine-readable API? How much of it should be mandatory? Can we relax some aspects to be just optional?


Question: úložiště - zobrazení struktury ve stromu?

No ona to nebyla vteřina, bez refreshe nám to nešlo. Ale dneska už je to OK, tak nevím, jestli už jsi s tím něco dělal, nebo se ProArc jen potřeboval prospat.

úložiště - zobrazení struktury ve stromu
V úložišti v panelu se zobrazením Tabulka a Mřížka by se hodila i možnost zobrazení ve stromu, aby se nemuselo postupně proklikávat po úrovních.

Vyberu si ve stromu číslo periodika, udělám změnu v popisu strany, dám uložit - a strom se mi sbalí. Nejde to udělat tak, aby zůstal rozbalený, jak byl?

Ještě se zeptám. Je technicky možné to potom předělat tak, aby ten strom nefungoval jen jako informativní zobrazení té struktury, ale že by se v něm dalo pracovat? Aby měl stejnou lištu jako tabulka a šlo v něm třeba vytvářet a přesouvat objekty (typicky ročníky a čísla u periodik)? Jen se ptám, jestli by to vůbec šlo.

Pujde to. Ale nebude to okamzite. Ted jsem pouzil strom, ktery mame od zacatku.
Pro implementaci vsech tech funkcionalit bych mel vytvorit uplne nove. Ale, slo by to :)

Tak to je super. Ono se to bude hodit mj. pro zjednodušení zakládání nových ročníků/čísel/příloh periodik (kde to nejde udělat přes "vytvořit více") a rozřazování stran do objektů... že už by pak nebylo nutné pracovat ve dvou záložkách.
Mám na to zakládat nové issue, nebo budeme pokračovat tady?

Klidne tady. Muzeme definovat vsechno co bude umet ten strom.

Teď jsem narazila na další věc. Jsem v periodiku, mám otevřené různé panely vč. stromu. Udělám na úrovni ročníku změnu v metadatech, uložím. Změna se mi ukáže okamžitě v tabulce, mřížce... Ale ve stromu se ukáže jen v případě, že v tom panelu mám označenou právě tu úroveň, kterou měním. Pokud mám tam označenou jinou úroveň, musí dát refresh, aby se změna ukázala i ve stromu.
strom_akt1
strom_akt2

@SmejkalovaAnna Prosím, koukni se na tuhle novou funkci - díky.

Oprava metadat už se propíše hned i do stromu, to už je OK.
Ale ještě se strom neaktualizuje, pokud přesunu sken na jinou pozici. Abych to ve stromu viděla, musím dát refresh.

To bych probral na schuzce... Me se nelibi, a myslim si, ze bude delat problemy pravem se stromem stavajici zpusob chovani: menime poradi, ale jen docasne nez ulozime zmeny. Pro strom je to problem, proto, ze pri kliknuti na polozku stazneme data z jadra, a tam nejsou zmeny. To je duvod proc jsem pri tahani objektu do stromu hned se objevi hlasku pro podtvrzeni zmen.

Trochu nás překvapilo, že nastavení sloupců stromu, které mají být vidět, funguje jen v Hledat, ale u stromu v panelu ne. Nedalo by se to přebírat i sem? Nebo to bude potřeba řešit jako extra nastavení?

tohle bychom určitě měli řešit minimálně při rozřazování stran - viz URL

Prosím, proberete to na schůzce 18.3.? Dávám sem štítek redesign a odebírám k testu.

@luckajirku @albertoh Navrhuji tohle řešit společně s issue URL - do rozvoje bychom to zahrnuli jako společný bod něcoi ve smyslu rozvoje funkcí stromového zobrazení v klientovi. Souhlasíte?

Domluva na schůzce 8.10.2024: zavíráme, budeme zakládat issues na jednotlivé požadavky úpravy chování stromu.

Řešení ukládání stran do různých objektů - rozřazování do čísel per. apod.
Současné řešení ukládání stran do jiných objektů klikáním na Přesunout dost zdržuje. Ta fce je fajn, když člověk potřebuje jen výjimečně přesunout pár špatně umístěných stran, ale třeba na rozukládání celého ročníku periodika do čísel to není.
Ve starém člověk přepnul do editoru, tam měl rozbalený strom toho periodika a postupně uložil vše do jednotlivých čísel. Přitom krásně viděl celé per. a mohl si to kontrolovat, strom zůstával rozbalený. Jak by to šlo řešit tady? Současné řešení je nepoužitelné - resp. je to pomalé a méně přehledné - a spíš se takhle udělá chyba, že se uloží ty strany jinam než patří.
Mrkněte na to prosím vás z jiných knihoven, já si moc nedovedu představit takhle dělat nějakou větší dávku.

@nezbedova @SmejkalovaAnna Prosím, mohly byste se na tohle podívat z pohledu rutinního provozu - periodika, stt přívazky ... ? Moc děkuji.

Já se na to podívám, až to bude zase fungovat, teď jsem zadávala issue URL nicméně @luckajirku Lucko nestačilo by jen to vyřešit tak, kdyby při dalším otevřením okna (klepnutím na šipičku) tam zůstalo naposledy otevřený (resp. cílový objekt) v rozbaleném stromě? Jako dřív?

To je ale jenom část. Jedna věc je, že si to má pamatovat, kam jsem naposledy přesouvala. Druhá pak, že pokud přesouvám třeba 300 stran do 50 výtisků nebo příloh, potřebuju mít tu agendu otevřenou, dokud nepřesunu poslední strany. A ne padesátkrát klikat na přesouvací ikonu. Vyskakovací okno dává smysl, pokud se dá pracovat jen v něm - pokud musím překlikávat, tak je to zdlouhavější než ve starém, kde se to dělalo po přepnutí do editoru nadřazených objektů.
@albertoh, nedalo by se to vyskakovací okně rozdělit (stejně jako u fce Označit řadu), abychom tam pak měli v levém sloupci otevřený ten původní objekt s uloženou celou dávkou, a až ve druhém sloupci pak hledání v úložišti? A v levém bych si to označila, v pravém našla, kam to chci uložit, uložila, zaktualizovaly by se automaticky oba sloupce, zas bych vlevo označila další strany, vpravo bych naklikla další výtisk atd. A okno by se zavřelo až po posledním přesunu.

A tohle je ještě ten jednodušší případ, kdy jde čísla vytvořit přes více a nachystat si je předem - a v tomhle kroku se "jen" rozřazuje.
Pak je ten případ, kdy se čísla zakládají jednotlivě a je potřeba mít k tomu skeny a opisovat údaje z nich - to se teď dělá ve dvou záložkách (v jedné otevřené skeny, ve druhé periodikum - struktura + metadata) a bylo by fajn najít způsob, jak to moci dělat v jedné obrazovce a hned ukládat do nově vytvořeného objektu i ty skeny.

Nasadil jsem novou verzi kde je mozne menit nadrazeneho pomoci pretahovani objektu z panelu vazeb do panelu se stromem.
Pri pohybu se mozne cile obarvy zelene nebo cervene podle povolenych vazeb. (@pkudela udela peknejsi disajn, zatim jde o funkcnosti)

image

 Zaroven, dialog zmeny nadrazeneho objektu jsem zmenil tak, ze lze hledat vsechny modely, ale tlacitko "presunout" je aktivni jen pri vyberu objektu s povolenou vazbou

image

image

To hledání a (ne)umožnění uložení podle vazeb je super.
Ale to rozukládávání do čísel je pořád nevyřešené.

Rozukladavani jsem resil pomoci tahani z panelu vazeb na strom

On problém je v tom, že ten strom zrcadlí tabulku/náhledy..., kdežto ten editor nadřazených objektů ve starém umožňoval mít to PŘEHLEDNĚ otevřené na jiné úrovni.
A je otázka, nakolik je to tahání myší dobré (je to rychlé, ale zas snadněji se strany uloží jinam, než patří) - k tomu už kdysi probíhala diskuse:-).

Předělat dialog ukládání do objektu tak, aby zůstal označený poslední objekt, do kterého bylo uloženo. Pokud ukládám poprvé není označené nic, ale musím objekt vyhledat.

Toto je opravdu potřeba dodělat - pro instituce, které dělají periodika, je to dost zásadní.

Na schůzce 15.3. potvrzeno, že zadání je v této fázi postačující.

Predelal jsem chovani dialogu pro vyberu nadrazeneho objekta.
Ted si pamatuje posledni pouzivany, a pri otevreni se automaticky vybere.

@albertoh pořád to není ono:-). Člověk musí označit strany, kliknout na přesunout atd. a pak zase označit další, zase kliknout na přesunout... Zbytečné klikání - a pro rozukládávání velkého množství stran je to nepoužitelné. Už jsem to psala tady URL

@albertoh je k tomu od nás potřeba ještě něco dovysvětlit, ukázat, jak se to používá atd.? a bude to v té verzi, která bude od pondělí k testu, už dořešené?

Ne ne, uz na tom delam.
Jakmile budu mit funkcionalitu, predam Petrovi aby to dal do krasy.
Kazdopadne, pracuju na tom tak, ze nove, ten dialog bude mit tri casti (jako Oznacit radu):
 - vlevo seznam objektu na presouvani v tabulce
 - uprostred hledani
 - vpravo strom hledaneho objekta
 -
Pri posouvani objektu zustavame tam, aby se dalo vybirat dalsi v leve tabulce.

Je to ono?

možná bych to radši zkoukla dřív, než se to "dá do krásy", ať se to kdyžtak nečeše zbytečně:-). nevím, jestli to dobře chápu.
teď v tom vyskakovacím okně bude ve druhém a třetím sloupci to, co je teď ve starém pod sebou? uprostřed bude jen výběr titulu a vpravo bude strom toho vybraného?

Ošklivé je v pohodě, jen je tam dost cestování po obrazovce. Označím vlevo, najdu uprostřed, pak musím úplně doprava, tam si označím, pak zas úplně doleva potvrdit přesun. Ve starém bylo celé to hledání vč. stromu pod sebou. Málokdy se jedna dávka rozřazuje do různých titulů, takže tu prostřední část využiješ na začátku a pak už ne. No, dá se to asi řešit tak, že si vyhledám titul a pak ten prostřední sloupec zúžím tak, že už nepřekáží, ale zas tím přijdu o takovou tu ujišťovací pohledovou kontrolu, že jsem ve správném titulu...

Takze radsi vlevo vyber objektu k presouvani, a vpravo hledani a pod nim cil?

Tak ještě něco - chybí tam náhled. Takže přeci jen by tam měl být třetí sloupec, ale v něm by měl být náhled (poslední nakliknuté) stránky. (Stejně jako to je v tom okně pro Označit řadu.)

A další věc - bylo by možné, aby v hlavičce toho prvního sloupce bylo vidět, odkud se to přesouvá? Ve starém vidím číslo a datum vydání - ideální by teda bylo, kdyby tam byl vidět i titul. Šlo by to tam přidat? Aby tam bylo něco takového, jak je to na liště v Hledat, když se proklikávám níž:
hlavickazdroj

@albertoh

`Málokdy se jedna dávka rozřazuje do různých titulů, takže tu prostřední část využiješ na začátku a pak už ne.`

A u nás z jedné dávky STT rozdělujeme do různých titulů (i 80 adligátů), takže jsem to chtěla vyzkoušet, ale nejde mi to. Naskočí tabulka a točí se kolečko. Nevím, zda postupuji správně.

@luckajirku jak se ma prosim jmenovat ten dialog? "Vyberte nadrazeny" nebo "Vyberte nadrazeny objekt" nebo neco jineho?

<img width="302" alt="image" src=" URL

@pkudela Mám problém při testování. Po zavření okna mám mlhu.

obrazek

A když jsem chtěla připojit soubory po novém importu k titulu pomocí tlačítka Pokračovat, tak mám chybu. A nevídím Orázek.

@albertoh Netestuji brzy? Neděláte na tom ještě?

@albertoh Zkusila jsem. Mám i nové okno i anonymní okno. Po pokusu o změnu nastavení rozložení oken pro editaci mám mlhu. Musím dát refresh nebo kliknout jinam (např. Správa dávek). Pak se vrátím/pokračuji k metadatovému popisu a to už vidím zobrazení jak jsem to nastavila. obrázek nevidím ani při editaci ani při přesunu.
Po Pokračovat mám chybu.
obrazek

Chrom i Mozila.

@albertoh Obrázek vidím, ale nemohu připojit do objektu.

obrazek

@nezbedova @luckajirku Prosím, máme chtít s tímhle omezením vydat verzi klienta? Osobně bych byla pro ji vydat a tento týden testovat podle plánu. Děkuji.

@ZdenkaSera za nás je to v pořádku, i ve starém se celá dávka ukládá na jedno místo - a to, jestli by to šlo rozřazovat už třeba v importu, a jak přesně by to mělo fungovat, to je na další vývoj.

Ne, toto v současnosti není možné a bylo by to velmi komplikované to dovolit už v importu.

Buďto je možnost si to rozřadit ještě před importem nebo to udělat až když je uloženo do aplikace.

Tak pokud tohle není příprava na tu možnost už v importu, tak by se to mělo chovat a vypadat asi malinko jinak - jestli tam jdu z importu přes Pokračovat, nebo jestli rozukládávám v úložišti přes Přesunout. V importu by měla být jiná obrazovka, kde nebude Přesunout, ale Uložit dávku atd.
Jinak teď nevím, kam to psát, protože se to týká toho prvotního uložení z importu, ale teď je to tak divně spojené tím stejným vyskakovacím oknem. Narazili jsme ještě na to, že když dám uložit/přesunout tu dávku v importu a po uložení dám Otevřít v editoru, tak se to pod tím otevře, ale to vyskakovací okno tam zůstává viset. Zavřít ho můžu, ale pokud volím možnost otevřít v editoru, očekávám, že se mi tohle zavře a ocitnu se hned v tom objektu v editoru.

@luckajirku @nezbedova Můžeme se v tom případě domluvit, že tohle bude k diskusi na schůzce 18.4.?

Děkuji za reakce.

@albertoh Prosím, vydáš tedy během dneška verzi, abychom mohli nasadit s novým jádrem? Děkuji.

Nejde vydat release. Asi github ma problemy. Zkusim jeste pozdeji

Třeba se github kousl, protože tuší, že je tam potřeba ještě nějaké věci opravit:-)

V základu to funguje dobře, ale měl by se po uložení stran zaktualizovat strom. Uživatel by měl mít možnost si hned zkontrolovat, že se tam ty strany přesunuly. Ve starém po přesunu sice ten editor s uložením zmizí, ale po vybrání další strany se tam objeví a přesunuté strany jsou už vidět na novém místě. V klientovi zmizí ze starého umístění, ale v novém vidět nejsou.

Souhlasím s @luckajirku. Při rozřazení např. do čísel periodik je třeba kontrolovat, jestli se to přesunulo správně a logičtější je to dělat přímo v tom vyskakovacím okně. Jinak ostatní chování otestováno a je OK.

@albertoh prosím tě, jak tohle vidíš? vím, že jsi v tom issue pro strom psal, že je problém s aktualizací, pokud se přesune objekt v tabulce a ještě není uložená změna, ale tady k uložení změny už došlo, takže to je trochu jiný případ a problém by tam být neměl, ne?

Tenhle je jiny strom. On neovlivnuje veci editoru. Tady to udelam. Ale ne tenhle tyden, jak jsme se domluvili

Pořád ne:-). Strom se obnovuje, ale po uložení není vybraný poslední objekt, do kterého se přesouvalo. Pokud okno zavřu a znovu otevřu, pak se to posune na ten správný, ale jinak si to drží ten poslední objekt z předchozího použití fce Přesunout. Potřebujeme, aby se to posouvalo na poslední vybraný objekt už během toho jednoho rozukládávání, jako to funguje ve starém.

Jo, tohle už je ono, aktualizuje se to správně a označuje se to, co má.
Jen ta hláška asi musí ještě dozrát... Zatím se ukazuje tohle:
editor.children.relocate_dialog.titleRepo
editor.children.relocate_dialog.messageRepo

tak už se nám to zase rozbilo - asi s tou opravou posouvání lišty. Když přesunu strany, uložím a dám zavřít, tak mám v úložišti krásně šedé prázdno - až po F5 se obsah načte.

Funguje to dal. Akorat je tam vterina v sede. Nesouvisi to s tim posouvanim.
Stava se kdyz mam editor metadat. Hledam pricinu

Zatim nic nedelal, ale chci to zlepsit. Ten editor metadat ted se nacita hodne dlouho. To je kvuli to rozbaleni...

Ted se pri obnoveni objevuje "loading bar" nahore, aby bylo jasne, ze obrazovka neztuhla, jen dumaaa :)


Question: Support arrays in generated equals function?

Or if not, we could make it `compileOnly` for non-native targets only, I imagine.

Annoying that end-consumers will have to download the Poko deps for no reason, but I agree it's not a deal-breaker and can be a future optimization. Closing!

Support arrays in generated equals function
`toString` and `hashCode` already support this. `data class`es do not, and have an IDE warning about it.

It should be fairly trivial to add support, I'm just not sure if it's a good idea to. It may be unexpected behavior?

Would you consider making it opt-in with some mechanism? Probably with an annotation like `@ByContent` or `@ContentBased` or something?

Yeah I'll consider it. I've held on to a "just do what data classes do except for the one thing" approach for this long, but there's an ever-growing list of interesting features to consider adding so maybe it's time to bite that bullet.

One small challenge is that I currently include the `poko-annotations` dep by default, but allow substituting a custom annotation. If I add more annotations, do I want to allow them to all be customizable, or just drop that feature and force the default annotations dep? Probably the latter but that's a breaking change.

Is Redwood using arrays for performance or another reason? I had the thought of wrapping your arrays in value classes with custom `equals` overrides, but that isn't supported yet and may be syntactically annoying for your use case.

It is for performance, yes. Some of our code runs inside a JS engine where `Array` translates directly to JS's native array whereas `List` is a few Kotlin classes wrapped around an `Array` which adds a lot of copies for things like expansion. Since the only types which interact with these arrays are generated classes, we don't need to worry about the ergonomics as much.

So Poko being JVM-only is probably going to block your adoption too? 😬

Well, yes. But that's fixable with some PRs!

Wanted to make sure use cases were covered first. Although we only need this for two or _maybe_ three types so writing equals/hashCode manually for them wouldn't have been the end of the world.

Kotlin's decision to avoid special array-handling in data class equals functions is for JVM-specific reasons. Is it the case that JS native arrays are also not compared by content? Or are the Redwood classes targeting JVM along with JS?

Correct. They mirror the JVM semantics. You must use contentEquals/HashCode/ToString to get the contents rather than the array reference to be used.

We target JVM, JS, and many native targets with this code, yes. The efficiency gains are paltry on non-JS platforms.

Some notes as I dive deeper – everything I mention here for data classes is also true for Poko classes:
* Data classes do consider content for array-typed properties when computing `hashCode` and `toString`. `equals` is the only one for which array content is ignored.
* Data classes do *not* consider content for `Any`-typed properties that are arrays at runtime when computing `hashCode` and `toString`, so Poko would need to implement its own handling for this.
* Data class handling of `hashCode` and `toString` for arrays is specifically built in to the IR compiler. Poko would need its own `arrayMemberEqualsSymbol`, and figure out how to map that to the platform-specific `Array.equals` equivalent (I think this is called "lowering"?).
* Need to decide whether to handle nested array types (`Array >`) – that would be `Arrays.deepEquals` on JVM, and the `hashCode` and `toString` implementations would need to be updated to do this too since they currently don't handle nested arrays. Incidentally, `deepEquals` *does* handle Array members even if they are typed `Any`.
* Kotlin's stdlib has `contentEquals` and friends defined in common code, so mayyybe this can be referenced without any platform-specific handling?

I added #135 to test the current handling of arrays in the sample project.

Confirmed that calling common-code stdlib functions from generated IR code is supported, which simplifies this a lot. 🎉

I plan to:
1. Add the new annotation as an experimental API.
2. Implement `contentEquals` for all typed and primitive array types.
3. Handle `contentDeepEquals` for nested array types, updating `hashCode` and `toString` accordingly.
4. Handle `Any` properties that are arrays at runtime.
5. Flesh out the path forward for custom annotation consumers – probably deprecate this feature.

Simple array content support is enabled in the latest `0.14.0-SNAPSHOT`, implemented in #141. Not sure if it's possible for Redwood to try it out on just the JVM, otherwise I filed #142 for multiplatform support.

Support multiplatform
I never really knew what `data class` did on other platforms, but it sounds like they generate the exact same functions that they do on the JVM. Since Poko is implemented in IR, it should theoretically "just work" for other platform targets.

Haven't done much with Kotlin multiplatform so help and/or PRs are welcome.

Steps to support multiplatform:
- [x] Update annotation module structure to multiplatform
- [x] Update Gradle plugin to apply to all platforms
- [x] Add compiler tests for additional platforms
- [x] Create sample project(s) for additional platform(s)
- [x] Update CI to build and test all platforms
- [x] Determine best use of `implementation` vs. `compileOnly`
- [x] Update CI to publish all platforms

Oh yeah, forgot I switched to the publish plugin that does that for free


Question: ParseOptions: Add `read_tags`?

Kid3 puts markers to show that you can add another tag to the file, not necessarily that it's present, BTW.

Yeah, I haven't actually seen anyone make use of TagLib's `StringHandler`. I imagine in most cases no errors are noticed since Picard (and likely other tools) add an ID3v2 tag for compatibility. The real issue here is that in Lofty it's a hard error, rather than a discard & warn.

ParseOptions: Add `read_tags`
### Summary

Unexpectedly, many projects pull in Lofty *solely* for its property reading. Each time it is used for this purpose, however, all of the tags are still being parsed unnecessarily.

Just as one can skip property reading with `ParseOptions::read_properties(false)`, it may be worth adding `ParseOptions::read_tags()`.

### API design

rust
impl ParseOptions {
 pub fn read_tags(&mut self, read_tags: bool) -> Self;
}

hello, it would be great to be able to do `ParseOptions::read_tags()`. Or wouldn't we have to use a [features] for this? Because in my case I wouldn't need the tags at all.

@vincehi Hello!

There wouldn't need to be a feature for this. Since `read_properties` already exists, this would act the same. When parsing, the tags will just be skipped and stored as their defaults (`Id3v2Tag::default()`, etc.).

Does this seem complicated to you? I preferred to use taglib even though it also analyses tags even though I don't need it, but it doesn't crash on the file format (UTF-...). On the other hand, the length recovered is in milisecond on lofty, which is very convenient for me.

No, it'd be a pretty quick feature to add. I just haven't bothered since I've been working on more requested features.

Can you make issues for any crashes you have with Lofty? There haven't been any text encoding issues that I'm aware of.

I think I had the same problem as this person URL But if I understand correctly, it's on the read tags, not on the read properties. So if in future I could bypass the read tags that would be great.

Yeah, this would avoid the error in #373 entirely. I'll see about getting this in 0.21.0.

Wav: Failed to read RIFF INFO item value
### Reproducer

I tried this code:

rust
let tagged_file: Option = match lofty::read_from_path(
 "C:\\Users\\ferry\\Music\\test\\大哉乾元(洛天依人声版)_MMM.wav",
 ) {
 Ok(value) => Some(value),
 Err(err) => {
 // Wav: Failed to read RIFF INFO item value
 println!("{}", err);
 None
 }
 };

The lofty version is "0.18.2"
image

### Summary

There is an error "Wav: Failed to read RIFF INFO item value"

The Windows Properties page can read the tags in it. I can also play it on Windows Media Player.

Can anyone provide some information about this? Thanks.

### Expected behavior

Read tag correctly.

### Assets

大哉乾元(洛天依人声版)_MMM.wav

Hello!

Your file has items that are not UTF-8 encoded. What is your local character encoding? If Windows is able to read it, that means the file is using your system encoding.

UTF-16.
And the file's title, artist, ... tag are UTF-16 LE encoded.

Hm. It's unfortunate that RIFF doesn't specify a text encoding. We would be able to *somewhat* reliably detect UTF-16 LE, but there's no BOM either. The only way for us to know ahead of time how to decode the text would be using GetACP() from Windows. Don't know how I feel about having platform-specific code in Lofty, though.

Maybe we can simply set the way to decode text when read riff info. Obviously it is not a good way to fix it though.

That would also work. Gonna have to think about this one. For now I'd just recommend changing the encoding of your files to UTF-8, that's what most libraries (outside of TagLib) will exclusively support.

Being able to use alternate encodings would be nice. In my project I need to build an index of all tag data (title/artist/genre/etc). I'm guessing these files are in UTF16 so I'm having the same issue.

They are able to be read in KDE Elisa. I think that project uses taglib though.

Do your files have other tags? In the case of the file provided in this issue, it has both a RIFF INFO and an ID3v2 tag. Elisa ends up just using the data from the ID3v2 tag and logs UTF-8 errors for the RIFF INFO tag.

I added metadata to several .wav files with musicbrainz picard. I think picard adds ID3v2 tags and RIFF Info tags to wavs by default. How would I check the tag types?

Running `ffprobe` against it shows the metadata with a warning simlar to: `[json @ 0x5664cd0010c0] 1 invalid UTF-8 sequence(s) found in string 'Example�s', replaced with '�'`. The missing character should be an apostrophe.

I have another file with the ellipses character. It also fails in lofty. No errors in ffprobe or elisa.

I have other files with non-english characters. They play in Elisa and I can view the metadata correctly there. When I pass the files into lofty, I get the following error: `FileDecoding(Wav: "Failed to read RIFF INFO item value")`

That's probably it. Most serious music players will check all tags to fill in any missing fields. Trying to load a non UTF-8 RIFF INFO tag with Elisa will log errors, and it will fallback to the info it can get from the ID3v2 tag. Elisa just depends on `KFileMetaData` which uses TagLib, and as far as I can tell they don't set a `StringHandler`.

I will say, a hard error shouldn't be the case outside of `ParsingMode::Strict`. That's something I can fix (just ignore it and log a warning, like TagLib).

I use kid3 to see what tags a file has. It uses TagLib, and with the asset provided in the issue, it ignores any field in the RIFF INFO tag that it can't parse as UTF-8.

I downloaded the kid3 program. The files that failed to read in lofy had an ID3 **and** a RIFF tag. Some also had extra tags, but it didn't list a type or any info.

I noticed some files display the data correctly in the ID3 tag, but show an empty entry in the RIFF tag. For example, the file with the invalid UTF8 warning when inspected with ffprobe had an apostrophe in the title. It is visible in the ID3 tag, but the RIFF tag shows the title as empty (key exists, no value). Other values were replaced instead. ID3 has a - (dash) in the artist entry while RIFF has a ? (question mark).

From above, I think in the future, `ParsingMode::Strict = false` will cause `read_from` to log Err instead of returning Err when encountering a tag that cannot be parsed correctly. If at least one tag is parsed properly, read_from will still return OK. Otherwise, the Err will be returned.

Interesting! I've never heard of this app, but I think it will be helpful for my project. I was using picard to inspect metadata but this one seems better for what I need.


Question: Container crashing when trying to retrieve VPN configs?

Exactly, the clones repo is stored in config so check that the git clone
deleted

On Wed, 27 Sep 2023 at 07:10, Niels Morf ***@***.***> wrote:

Container crashing when trying to retrieve VPN configs
### Is there a pinned issue for this?

- [X] I have read the pinned issues

### Is there an existing or similar issue for this?

- [X] I have searched the existing issues

### Is there any comment in the documentation for this?

- [X] I have read the documentation, especially the FAQ and Troubleshooting parts

### Is this related to the container/transmission?

- [X] I have checked the container repo for issues

### Are you using the latest release?

- [X] I am using the latest release

### Have you tried using the dev branch latest?

- [X] I have tried using dev branch

### Config used

yaml
version: '3.8'
services:
 transmission-openvpn:
 container_name: 'haugene'
 cap_add:
 - NET_ADMIN
 devices:
 - '/dev/net/tun'
 volumes:
 - /volume1/data/transmission-data/:/data
 - /volume1/data/haugene/resolv.conf:/etc/resolv.conf
 - /volume1/data/haugene/:/config
 environment:
 - OPENVPN_PROVIDER=PROTONVPN
 - OPENVPN_CONFIG=dk.protonvpn.net.udp
 - OPENVPN_USERNAME=**None**
 - OPENVPN_PASSWORD=**None**
 - LOCAL_NETWORK=192.168.1.0/24
 - OVERRIDE_DNS_1=94.140.14.14
 - OVERRIDE_DNS_2=149.112.112.112
 - OPENVPN_OPTS=--inactive 3600 --ping 10 --ping-exit 60
 #- TRANSMISSION_RATIO_LIMIT=3
 #- TRANSMISSION_RATIO_LIMIT_ENABLED=true
 - TRANSMISSION_SPEED_LIMIT_UP_ENABLED=true
 - TRANSMISSION_SPEED_LIMIT_UP=1000
 logging:
 driver: json-file
 options:
 max-size: 10m
 sysctls:
 - net.ipv6.conf.all.disable_ipv6=0
 ports:
 - '9091:9091'
 image: haugene/transmission-openvpn:dev

networks:
 default:
 external:
 name: mybridge

### Current Behavior

Container crashes at start with this error:

The issue started somewhere in the last 8 days, it was working fine 8 days ago and as far as I can tell ourobouros did not upgrade this container in the last 8 days.

### Expected Behavior

container starting normally

### How have you tried to solve the problem?

1. tried the dev channel
2. DNS seems to work because I can see this in the logs:

### Log output

Starting container with revision: 52d432ddca774080040627e3b6ec61fc9e6b0ac7
TRANSMISSION_HOME is currently set to: /config/transmission-home
One or more OVERRIDE_DNS addresses found. Will use them to overwrite /etc/resolv.conf
Creating TUN device /dev/net/tun
Using OpenVPN provider: PROTONVPN
Running with VPN_CONFIG_SOURCE auto
No bundled config script found for PROTONVPN. Defaulting to external config
Will get configs from URL
Repository is already cloned, checking for update
error: Your local changes to the following files would be overwritten by merge:
 openvpn/privado/updateConfigs.sh
Please commit your changes or stash them before you merge.
Aborting
Updating 45cc9b305..35ec20293

### Environment

markdown
- OS: DSM 7.2
- Docker: 20.10.23

### Anything else?

_No response_

I erased the content of the `vpn-configs-contrib` folder to test, and the error is now:

I also tried removing `OVERRIDE_DNS_1` and `OVERRIDE_DNS_2`, it doesn't seem to change much.

URL
delete vpn-configs-contrib
then delete container and restart..
it's a git thing..

Ha. I had to delete the entire `vpn-configs-contrib` folder. I had previously deleted the contents only but not the folder itself, which was somehow not helping. Maybe some hidden files I missed. Thanks.

openvpn/privado loop
### Is there a pinned issue for this?

- [X] I have read the pinned issues and could not find my issue

### Is there an existing or similar issue/discussion for this?

- [X] I have searched the existing issues
- [X] I have searched the existing discussions

### Is there any comment in the documentation for this?

- [X] I have read the documentation, especially the FAQ and Troubleshooting parts

### Is this related to a provider?

- [X] I have checked the provider repo for issues
- [X] My issue is NOT related to a provider

### Are you using the latest release?

- [X] I am using the latest release

### Have you tried using the dev branch latest?

- [ ] I have tried using dev branch

### Docker run config used

version: '3.3'
services:
 transmission-openvpn:
 cap_add:
 - NET_ADMIN
 volumes:
 - '/mnt/downloads:/data'
 - '/opt/transmission/config:/config'
 sysctls:
 - "net.ipv6.conf.all.disable_ipv6=0"
 environment:
 - OPENVPN_PROVIDER=HIDEME
 - OPENVPN_CONFIG=UK
 - OPENVPN_USERNAME=************
 - OPENVPN_PASSWORD=************
 - LOCAL_NETWORK=192.168.1.0/24
 logging:
 driver: json-file
 options:
 max-size: 10m
 ports:
 - '9091:9091'
 image: haugene/transmission-openvpn

### Current Behavior

I am not using Privado, I am using Hide.me. Upon start, the container starts and shows port in list. The port disappears once the container goes from starting to running status. I check the log and I get a loop then it aborts as seen in the logfile attached below.

### Expected Behavior

All functions load and are accessible and transmission connected behind Hide.me VPN

### How have you tried to solve the problem?

Restarted container
Stopped & restarted VPN
Rebooted machine where container is housed

### Log output

_transmission-openvpn-transmission-openvpn-1_logs.txt

### HW/SW Environment

markdown
- OS:Unbuntu 22.04.2
- Docker:24.0.5, build ced0996

### Anything else?

_No response_

That put me in the same state... still on the loop of a VPN i dont even use

If you deleted the container completely it wouldn’t have a git cache of
downloaded profiles..

On Tue, 26 Sep 2023 at 22:10, Xsatc77 ***@***.***> wrote:

@Xsatc77 if you delete the config from the opt/transmission/config directory the configuration gets freshly cloned and the issue disapears :)
You have mapped the config as volume and therfore just recreating the container does not work.