Question: Migrating/Integrating WebCrypto (possibly WASM) instead of Node Forge Crypto?

We decided not to bother with preventing resource starvation, however an idea is like this.

1. Take advantage of DB's natural key ordering.
2. Create a bimap index of Priority/Timestamp -> Task Id AND Timestamp/Priority -> Task Id
3. Now we can iterate task ids based on 2 compound indexes: highest priority + earliest timestamp AND earliest timestamp + highest priority
4. Use dynamic programming/kinetic priority function that iterates through both sublevels (indexes) simultaneously to fill up a fixed concurrency pool (if unlimited, this policy is unnecessary, just iterate through as fast as posssible)

Simultaneous iteration that uses the timestamp to weight the priority, where the timestamp delta starts from 0 and goes towards infinity. Once could say that this multiples the priority based on a "rate". A delta of 0 multiplies by 1. A delta of infinity multiplies by infinity. Therefore the rate produces a multiplier between 1 to infinity.

Migrating/Integrating WebCrypto (possibly WASM) instead of Node Forge Crypto
### Specification

The work in #155 involving nativescript meant that we are starting to consider alternatives to Node forge crypto in order to standardise our crypto API and future proof to future JS-based platforms. There are number of reasons to look into webcrypto:

* WebCrypto is a standard across ES platforms, thus enabling more potential native-implementations which can improve performance and security (prevent timing attacks)
* WebCrypto API focuses on using ES-compliant buffers like `Uint8Array` and `ArrayBuffer` which can help us standardise our buffer usage across js-db, js-id, js-workers and more, especially as workers require `ArrayBuffer` to do zero-copy
* There is existing work with WebCrypto API involving WASM to plug in functionality that isn't supported by the standard, such as providing ed25519 keys, and this will help us resolve the usage of ed25519 #168
* The `node-forge` source code isn't well maintained and doesn't have as many eyes watching and evaluating its security

Now for some background:

This comment URL explains how we came to be using `node-forge` as opposed to other cryptographic libraries.

I know that when Polykey project first started, we initially were thinking of using PGP and thus the kbpgp.js library ( URL We ended up going away from PGP due to its limited usage in a number of scenarios that we want PK to deal with. Namely end to end encrypted network communication which is a TLS issue, that makes use of X.509 certificates rather than PGP certificates. Furthermore we also had symmetric encryption/decryption scenarios like js-encryptedfs that again would not make use of PGP standards. Therefore it just lacked interoperability with many other cryptographic scenarios, it seemed like its own island of standards, the library can still be brought back in in the future if we find usecases for PK using PGP.

However in choosing node-forge, we came across a few other problems. Mainly overall-cross platform compatibility planning for mobile devices. This is not just a problem with crypto, but also other libraries that are used in our `networking` domain such as utp-native.

Here are something I found that may be relevant to us proceeding here:

* The webcrypto standard ( URL is now the official standard for cryptographic operations in browser environments. It is now being supported by all major browsers and electron
* This webcrypto standard is now taking over other deployment platforms such as in Nodejs: URL & URL
* As a standardised API, switching over to this API gives us more cross-platform opportunity as the rest of the world's development ecosystem catches up and migrates over to webcrypto standard.
* In particular are nativescript and react-native ecosystems. Currently none of them have official crypto APIs, they expect the developer to use underlying native iOS or Android crypto APIs.
* However I found out that the GUN project URL claims to be able to use webcrypto on react-native and other platforms.
* It does this through a "webview" trick. Basically this hooks into the browser runtimes that are on iOS and Android to perform the actual crypto and then return the results back to the main application. I have no idea about the performance characteristics of this trick. This trick is described here: URL and URL
* The usage of webcrypto libraries is pioneered by "PeculiarVentures" which has these main libraries:
 - URL - this could be an alternative to node-forge for all of our PKI/X.509 and TLS related functionality (previously it was claimed this was overly complex compared to `node-forge`)
 - URL - this is a generic polyfill for webcrypto, not entirely sure how it wraps the native webcrypto inside nodejs or if smooths over the differences, there is a discussion about this library in relation to other webcrypto polyfills targeting nodejs URL because this is also mentioned with respect to the webview trick, there may also be a relationship between the webview trick and this polyfill
 - They have many other crypto related libraries that we should investigate

All of this will mean that we either replace node-forge, or end up creating a adapter pattern where we plugin different crypto implementions depending on our environment. At this point in time, the `keys` domain abstracts over most(all?) crypto operations for all other domains. Except in the case of EFS which is currently pinned to `node-forge` (it may be a good idea to abstract that and expect an interface of functions for EFS).

Cross platform compatibility here isn't just about the fundamental crypto library. It's also about other parts of PK. One closely related situation is the JOSE libraries. As they involve cryptographic operations, they currently seem to "fix" their underlying crypto library as well. It would be ideal that if we standardise on a crypto library for cross-platform deployment, that we can also ensure that our JOSE library is using the same crypto library to reduce our crypto attack surface. We are currently using URL which uses native crypto depending on the platform including webcrypto. Contenders include URL (which fixes on node-forge) and URL

### Additional context

* URL - js-id also should be using WebCrypto
* #155 - will impact NativeScript deployment
* URL - can help make use of the migration to `ArrayBuffer` and typed arrays
* WebAssembly resources
 - There are 2 well supported runtimes for WASM:
 * wasm3 - supports interpretation
 * wasmer - supports JIT and AOT compilation
 - URL
 - URL
 - URL
 - URL - This describes how to use webcrypto together with webassembly when you need a custom crytographic functionality not supported inside webcrypto.
 - URL
 - URL and URL

Wasmer can compile WASM code to native code. But wasm3 is for interpretation. Why use interpetation?

It appears that in some cases interpretation can be more widely deployed. There are examples of iOS apps using wasm3. URL

It's becoming fast a standard target for many languages. Even TypeScript when ported to AssemblyScript can be compiled to WASM.

Once it is WASM, the only thing missing is broad adoption of WASI. If WASI is broadly adopted like it is in nodejs ( URL then pretty much we have a universal portable binary capable of doing relevant system operations. WASI is like a universal standard of system calls. Like a whole new POSIX standard.

Then one would just use WASM and WASI for all platforms.

### Tasks

1. ...
2. ...
3. ...

Note that `age` and wireguard both use URL

It's an alternative to AES GCM which has higher performance when the hardware doesn't support AES-NI instructions.

AES GCM is still good for compliance though, some places like government require this.

But the algorithm apparently works and is designed for software-only implementations.

We will need to elevate this issue to a higher priority, we now have empirical results on how long it takes to generate a root key in multiple arenas:

1. Tests
2. AWS fargate

On fargate, using the 0.25 CPU containers, it takes 20 minutes to generate a root key pair. Se can be seen by this cloudwatch cpu utilisation graph:

cpu

At the same time, key generation is using all cores by default. This doesn't even require the worker manager to be integrated, I think `node-forge` does this by default.

We believe that this is causing CPU starvation of all the other testing workers that jest creates and thus leading to test timeouts as can be seen in #394.

As we have deployed our testnet, and we expect the need to autoscale the agents, key generation is going to be an important workload, as it delay the scaling up process. Remember even if we pass in a recovery code, the keys must still be generated and this still takes time.

Furthermore we want to get #168 done before going to mainnet, because at that point users will be using our system, and we don't want to get stuck on legacy RSA.

It's possible to override the number of workers that the `generateKeyPair` `src/keys/utils.ts:64` uses. We can provide `workers: 1` as part of the options.

The `node-forge` has its own workers implementation it doesn't actually use `WorkerManager`. We are looking into refactoring `WorkerManager` with respect to the `Queue` and `Scheduler` implementation in #329. And replacing the crypto is important for our (second, our first one was back in around Jan - Mar 2021 which failed) beta launch, because we want users to be using #168.

Note that 16.17 has proper support for ed25519:

image

Current latest 22.05 is still 16.16.

The commit that introduced 16.17 into nodejs is 6e2536f1b09863e984f0479ea4b162e4fe86493d.

To use it, one must use `nodejs-16_x` as `nodejs` will be `18.9.0` which may have some major differences.

I'm going to try the latest master commit and see how we go. This will impact all of the nix-derived dependencies.

The benchmark results in #446 means that we do need webcrypto and due to iOS issues, we cannot use wasm. So we have moved to libsodium instead, and the benchmarks show that libsodium is 10x to 50x faster than webcrypto.

We are also going with native `Buffer`, and future usage will just use the `Buffer` polyfill.

There remains 2 uses of webcrypto:

1. bip39 recovery code
2. x509 signing

The first can be replaced with our own KDF mechanism. In that sense, we would no longer use bip39.

The second is a bit more difficult, it may be easier to create a webcrypto shim backed off libsodium and only for ed25519 signing.

Generic Non-Blocking Task Management ("Queue") for discovery and nodes domains
### Specification
Unattended discovery was added in #320, however, there is no concept of priority within the queue. There are three ways that a vertex (a node or identity) can be added to the discovery queue, and they should follow this order of priority:
1. Manually, via the discovery methods `queueDiscoveryByNode()` and `queueDiscoveryByIdentity()` (these are called in the commands `identities discover` (explicit discovery) and `identities trust` (explicitly setting a permission, so we want the Gestalt to be updated via discovery).
2. As a step in the discovery process whereby child vertices are added into the discovery queue in order to discover the entire connected gestalt.
3. Automatically by a process of rediscovery when we want to update existing Gestalts in the Gestalt Graph (to be addressed in #328).

Vertexes with a higher priority should be discovered first, either by being placed at the front of the queue or by modifying the traversal method of the queue. The priority queue could also be further optimised by grouping vertices from the same gestalt together when this is known (for example when adding child vertices).

### Additional context

* Original PR for implementing Unattended Discovery: URL
* Issue for re-adding discovered vertices into the queue to update the gestalt graph URL
* #188 - Indexing the LevelDB may be required to be able to implement a persistent priority queue
* URL - general indexing into js-db
* URL

### Tasks
1. Modify the existing Discovery Queue to be a Priority Queue
2. Ensure that when a user interactively wants to discover a gestalt vertex that it becomes the highest priority and gets executed first
3. Look into the potential for further optimising the priority queue, for example by having multiple points of comparison with varying levels of importance that can influence the priority of a particular vertex in the queue

There's a go implementation of persistent priority queue backed by leveldb here: URL It can be used as a reference for this.

Our priority queue needs to by default maintain order, because we do want to know the sorted list of jobs. But also allow us to add a special priority number on top.

From my imagination:

This reminds me of the indexing problem, where you can ask for a list of rows sorted by several columns. The first column would dictate the base sort, then subsequent columns would sort any ambiguous sub-orders.

Imagine we had 2 indexes. The first being your priority index using an arbitrary number, the second being the monotonic time index using `IdSortable`. You could sort on the priority index first, then sort on `IdSortable` second.

Maybe this then has a relationship to #188.

This can be done with a compound index. Prefix can be the priority number (lexinted), suffix can be IdSortable.

This means you can then stream results that are always ordered in terms of priority first then by time second.

Priority can start at 0 by default, and one can increment priorities depending on the origin of the task. Like tasks emitted by user wanting to lookup something can be set to a higher priority number.

We could do this directly by changing the queue domain key. But I'd suggest first solving the indexing issue in general first then building a compound index on top.

We discovered that the priority queue can also benefit from a uniqueness index creating a uniqueness constraint: URL

This means that duplicate tasks cannot go into the priority queue. Not entirely sure if this is required because a queue can still say they should process the same task over and over.

We should have a concurrency bound in the queue. This means how many tasks should be executed at the same time. By default unbounded meaning all tasks gets executed immediately without waiting to be done.

For IO bound tasks, you might as well have unbounded concurrency. For CPU-bound it can be sent to the web worker pool which is bounded by core count. Battery usage optimisation may also affect our limit too.

A generic Queue class has been implemented here: URL

This queue is not persistent, or a priority queue, however, it is designed to be a generic queue that can eventually be used in all places that require this functionality (including the Discovery Queue). The generic Queue can be refactored to meet this issue and URL at some point in the future, potentially incorporating the DB.

Renamed this issue to the general idea of non-blocking task management. It now has to solve for discovery, nodes management in terms of setting nodes, pinging nodes and garbage collection, as well as in relation to:

* #354
* #349
* #345
* #328
* #297

There's a relationship between the queue design and the `EventBus` system, as well as our `WorkerManager`.

Most important is for us to develop a `Task` abstraction. It can be a `class Task`, that represents a "lazy promise". Promises in JS are strictly evaluated, while these tasks will need to be lazily evaluated. Then our task management system can convert our lazy tasks to strict promises (which represents futures). More background info here: URL

Our task manager will need to have configurable:

* Concurrency limit - indicates the bound on the pool of currently executing tasks (can be 1 to unbounded/infinite)
* Executor - choosing to execute by Node's event-loop, or by passing it into the `WorkerManager` to be executed in a separate thread or core, the former should be used for IO-bound tasks, the latter should be used for CPU-bound tasks. This could be specified by the task creator, rather than the task manager itself.

Stretch goal is to also incorporate "time/calendar-scheduling" so that tasks can be executed at a point in time like cron.

Interaction between `EventBus` and task manager may be considered. The event bus is about communicating changes between domains, but the task manager is the one actually executing the tasks.

Tasks can be:

* Re-ordered or reprioritised or given priorities
* Can be observed for success or failure
* Can have errors handled
* Can be cancelled using our design for abort signal, and have their real side-effects cancelled
* Can be monitored for progress
* Can be persistent - backed by leveldb

This rabbit hole for this goes deep. So we should make sure not to feature creep our non-blocking task queuing needs.

Also to clarify, we are not creating a "generic distributed job queue", that's the realm of things like redis queue and URL There's so much of this already. We just need something in-process relative to Polykey.

Along with the configurable concurrency limit and executor, I think we should have an interface for the queue as well. Depending on the situation we may need just a simple queue, a priority queue, a persistent database queue like discovery uses, etc etc...
So far as the `Queue` cares it only needs to support `push` and `shift`. So we can make the `Queue` a generic class and pass it any implementation we want for storing the queue so long as it extends the interface.

It shouldn't be too hard to make the change. We just need to decide if this degree of control is desired. I can see a need for it though.

More prior work:

* URL

Actually the entire modern-async library is quite interesting, as it has implemented some of the things that we've been working on as well. But I think we won't just bring in that library, but instead extract parts out of it for our own implementation.

It has interesting ideas for promise cancellation as well, `Delayer`, `Scheduler` and `Queue` can all help. I believe that our usage of async generators and decorators might be more advanced though.

The library also exports a bunch of collection combinators that can work with asynchronous execution. So instead of say `Array.map` mapping a synchronous function, it could map an asynchronous function, and then wait for all of them to finish. Basically we do this with `Promise.all` atm. It also works with asynchronous iterables like async generators. This I feel is a different kind of thing, and I'd only want to bring in these utility functions where it's relevant. Maybe if JS had better treeshaking and package specific imports ( URL it would work well.

Additional resources...

Regarding delaying tasks (will be useful to understand how timeouts work and how we intend to persist them):

* URL
* URL
* URL
* URL

Regarding serialisation and persistence of lazy tasks (the persisted task in the Queue backing the DB isn't a promise, it's just some data on disk, but we may give back a lazy promise as an in-memory reference to the task):

* URL
* URL

We may not actually be serialising arbitrary code, but instead allow domains to register function callbacks into the queue, and then the queue will then call back these registered functions. That way the functions are somewhat dynamic but not just arbitrary code execution.

As for the lazy promise representation, make sure to take some ideas from URL I'm wary of bringing in such a huge library and API, so we should just take ideas from it and implement it in our own promise abstractions.

As for the worker dispatch, we want the workers to pull in tasks from the queue when done. However threadsjs (used by js-workers) doesn't expose a complex queue to use. So we won't be able to use their own queue, instead, we would want to augment our workers or augment the `Pool` class to take work from our own persisted queue. See: URL

During the testing of this it would be nice to incorporate URL ( URL into the model based checking of the queue functionality. Some background: URL

So I'm thinking that rather than queueing arbitrary code, you have to first register handlers like `queue.registerHandler(fName, f);` Then subsequently when you push a task like `queue.pushTask(task);` the end result is that when the task gets executed, it ends up executing a "static pointer" which is the `fName`.

But then this is actually really similar to `EventEmitter` and our `EventBus`. Basically registering handlers is the same as adding event handlers, then pushing tasks is similar to emitting events. See: URL

So there are similarities, there are some differences, our scheduling of tasks is not just a simple synchronous call, nor is it an asynchronous call as we have in `EventBus.emitAsync`.

The queue is also persistent unlike the eventbus, the actual emission of the event may be delayed by an arbitrary amount of time that is also persisted. There's a controlled concurrency to dispatching tasks.... etc.

So similar, but different. In that sense, there's no need to extend from `EventEmitter` (especially since we need to change that to event target)...

This URL has an interesting solution. They support static priorities. These priorities are set when an item is enqueued.

Once enqueued, the user gets back a ID.

There are 256 possible priorities from 0 to 255 inclusive.

When the item is stored, a key is derived from both the priority level and its ID.

 // Create new PriorityItem.
 item := &PriorityItem{
 ID: level.tail + 1,
 Priority: priority,
 Key: pq.generateKey(priority, level.tail+1),
 Value: value,
 }

As you can see here, the IDs are generated by the queue, and they are a counter that is simply incremented.

The resulting database KEY is likely a concatenation of the priority level and the ID. Assuming lexicographic encoding, there's just basically `PRIORITY + SEQUENCE`.

So then in the database, we can always find the highest priority with just the ordered keys, as it will be ordered by the priority first then the sequence second.

Interestingly, when dequeueing, it doesn't seem to consult the database, but some in-memory structure that indicates what is currently highest priority level to consider. But I wonder if this is necessary.

Surely it can just take a row off the database, as the database is the source of truth of what is highest priority.

The URL has 2 interesting features:

1. The ability to delay the dispatch of a job that is queued
2. The ability to stream dispatch jobs

The first feature appears to be done by passing a handler that determines when a job is valid to be executed

This handler is simply checked repeatedly whenever a job is iterated over.

The second feature is an asynchronous loop that just iterates over the jobs. If a job isn't valid, it just sticks it back into the queue.

In that sense it is really using it like a queue, but I don't think it's very efficient as if a job that isn't valid, it just repeating a busy loop with a delay until it checks again when the job is valid.

The concept of URL is interesting as it indicates the idea of priority that can grow or decay as a function of time. So then when an item is inserted, it's inserted at time `T` rather than with a priority. But there's a priority function associated. Therefore, to know the priority of the item, apply the function to `T`. This is an abstract interface, the implementation might be more complex.

But it is interesting because we can definitely store `T` into the database, but our indexing of the queue becomes more complex. In the case of the go solution, the priority is part of the index, so it's easy to find what is the lowest priority just by getting a sorted key-value.

One way to do this, that upon first starting the queue, we scan the contents of the queue which is an O(n) procedure. This gives us an understanding of what is in the queue, and then to create in-memory twins of this data. In particular this would be useful for any persisted delayed jobs, where we put down the time it was inserted, and the desired amount of delay.

The queue is not meant to be big, so O(n) at the beginning when loading the queue should be fine.

Now can this same mechanism be used to apply dynamic priorities, where because we know the time of the items in the queue, we can then re-sort them according to an in-memory priority? To do so, may seem to require some sorted partitioning of the jobs into their relative derived priorities.

The idea of scanning the priority queue database at launch is necessary because persisted start times with delays have no other way of being made available to the code.

Unless one were to index the database by scheduled time. Schedule time would be `INSERTION TIME + DELAY`.

Then the items would be indexed by this number.

A numerically sorted key-values, upon which popping the lowest number, would be the item that is earliest to be scheduled for execution.

At the same time, because the program knows its own time, we can attempt to pop tasks only within a given range search.

Imagine index of `INSERTION TIME + DELAY`.

Time is ticked 1 second at a time. Insertion time grows linearly.

Delay can be 0 or more.

As tasks are inserted at linear time steps `1+100, 2+50, 3+10, 4+0`, this fills up a queue of:

4
13
52
101

The dispatch, looks for all task where `index <= currentTime`.

Suppose current time is 32, then you get `[4, 13]` which are tasks that are due to be executed. In particular you have 2 tasks are overdue. But that's ok, this is best-effort execution.

But when does this dispatch execute? Is it just polling every second? The database is not capable of triggering things. It's state, not a process.

So dispatch may check at each time-tick, but this is also inefficient.

The dispatch can instead look for lowest indexed item. Then set a timeout to be executed based on the scheduled time in-memory. This is assuming all times are in the future. If they are in the past, then they should be dispatched now.

So suppose the above queue.

Upon the first lookup, it would see task 4. Assume current time to be at 2.

Then it would `setTimeout` for `2` to pop a task at time 4, which would find all tasks `index <= 4`.

However what happens if new tasks are entered into the system that should be executed even earlier. In that case, the insertion of new tasks should also trigger the dispatcher, but only if the new task `INSERTION TIME + DELAY` is earlier than task `4`.

So at any point in time, there will always be a `setTimeout` set to the earliest task that has to be executed. And this is reset every time a task is executed.

This avoids having to do an O(n) scan at the very beginning, and instead is more like an O(1) scan.

So that deals with tasks with delays and also no delays. No delays just mean a delay of 0.

What we have is a list of tasks ordered by "scheduled" execution times.

Static priority is just a matter of adding a numeric prefix. Starvation may occur here.

Dynamic priority means tasks should start with an initial priority. But then as we pop tasks out we must execute a function to figure out its priority.

Suppose there are 2 tasks, both are scheduled for immediate execution, that is a delay of 0. Which task should be executed assuming only 1 can be executed at a time?

The priority function is applied to both tasks, using the tasks's `INSERTION TIME` as a parameter.

The priority function will take into their account their initial priority, plus their dynamic priority returned by the function. This added up is weighed together, and the highest priority task is executed, the other task remains in the queue.

This means priority is not indexed at all, it's just a value. And we solve the starvation problem now.

We still need a `TaskId`, and if we use `IdSortable`, we pass in a `nodeId` into the `IdSortable` constructor. This is only used to to ensure that Ids can be generated uniquely and not conflict between different nodes. But this is only useful if the `TaskId` is going to share the same namespace.

Since this is not going to happen, it doesn't really matter, and we should make the `nodeId` optional.

Note that however, if the node ID changes, that should involve reassigning the `generateTaskId` function with the new node ID.

Note that since we are indexing by "scheduled time", we cannot just extract the time component of the `TaskId`.

So we have multiple sub levels here:

1. `['Queue', 'tasks']` - `TaskId -> Task`
2. `['Queue', 'time']` - `Time -> TaskId`

When creating a task, we create a task id, insert it with the task, then also assign a time to the task to be executed.

Found URL and URL and URL

We could replicate a little about the API design here.

When scheduling a new task. The task object is created.

This task object needs to represent a lazy promise.

A lazy promise in this case means that the promise doesn't mean that the execution has started. It's simply queued.

Because tasks are persisted into the DB, it's possible for the PK agent to be restarted, and one may wish to await for a given task ID. That means acquiring a promise for that task ID.

I'm thinking that we can lazily create a promise, that is one to one for each task. Then you can await this promise's result.

If multiple calls to acquiring this promise is made, the same promise is shared among all callers. This means a single `resolve` or `reject` call will send all awaiters.

Here's an example:

ts
async function main () {

 const p = new Promise((resolve, reject) => {
 setTimeout(() => {
 reject(new Error('oh no'))
 }, 500);
 setTimeout(resolve, 1000);
 });

 const f = async () => {
 await p;
 return 'f';
 };

 const g = async () => {
 await p;
 return 'g';
 };

 const r = await Promise.allSettled([
 f(),
 g()
 ]);

 console.log(r);

 // @ts-ignore
 console.log(r[0].reason === r[1].reason); // This is `true`

 // The same exception object is thrown to all awaiters

}

void main();

There are some constraints:

1. You cannot acquire a promise for a task that does not exist in the queue.
2. Creation of the promise may involve hitting the disk if the promise doesn't already exist relative to a task ID.
3. Creation of the promises has to be protected against race conditions with the object locking map pattern.
4. If you get a promise, it is guaranteed that within some finite amount of time that this promise will eventually resolve.

How is point 4 guaranteed? It is only possible to get a task promise in 2 ways:

1. During creation of a new task, the promise can be created afterwards.
2. One can ask `queue.getTask()`.

Now because it's a lazy promise, this could mean that the task is already executed by the time you ask for a promise for the task. This is only possible if the task is no longer in the queue (or is in some invalid state).

In this situation, when asking for the promise, the promise should be immediately rejected. Alternatively since the acquisition of this promise is lazy, one may throw an exception at this point. The point is, if you do get a promise, the promise must be settled eventually.

One of the initial ways to do this is to add an event listener for every task as soon as it is put into the queue. The problem with this is that now you get in-memory space complexity of O(n), where you have 1 listener for every task.

Listeners aren't always necessary, and maybe lots of tasks are put into the queue. In such a case, we can make the promise/listener itself lazy.

getTask(): Promise {
 // if we give you back a Task
 // you can be guaranteed to have it resolved or rejected
 // if we cannot give you a Task, because it's already executed, then we throw an exception at this point
}

Alternatively we do something like:

getTask(): Promise ;

And `undefined` means it's not possible to give you a promise to the task.

Another problem is that exactly is a `Task`? Is a data structure, or is it a promise? Maybe it's both?

The `Task` could be a class instance with enumerable properties, along with a method that allows one to acquire the promise?

I'm considering an API like this:

// true means that you are tracking the task immediately
const task = await queue.pushTask(lazy: true);

// Waits for the promise
await task;

// false means you are not tracking the task
const task = await queue.pushTask(lazy: false);

// Now it may result in an exception if the task is already executed
// This has to distinguish from the task itself being rejected
// ErrorTaskReference
// ErrorTaskRejected
await task;

Something like this means `Task` is in fact an extension of the `Promise`. Or at least a class that has the `then` method. It doesn't actually have to satisfy the entire promise interface. Which would include `.catch` and `.finally`.

Either way, the `lazy` boolean allows one to switch from a lazy promise to an eager promise.

In this sense, lazy simply means whether the task itself is being tracked or not. If it is being tracked, then `await task` is always guaranteed to either result in `ErrorTaskRejected` or the task's result. If it is not being tracked, then it only starts tracking when `await task` is called. Which calls the `then` method. At this point it may throw `ErrorTaskMissing` or `ErrorTaskRejected`... or anything else.

I may have something like:

ErrorTaskMissing - task itself is no longer around, it may already been fulfilled
ErrorTaskReference - task handler was not found
ErrorTaskRejected - task was rejected, see the cause chain

Is there a utility in having `ErrorTaskRejected`? Maybe only to create a set of possible exceptions, as the cause chain can have anything that the task handler itself throws. Otherwise we are just rethrowing the exceptions. URL

So within `Queue`, we will have 2 methods that that are used as part of `start` and `stop`.

protected async startProcessing(): Promise ;
protected async stopProcessing(): Promise ;

Their job is to peek into the job schedule. (I've started to realise that this is more a "schedule" not a queue, since the priority doesn't apply until the tasks are due for execution).

In the job schedule, they find:

* Tasks that are due for execution are dispatched the execution queue
* The next task not due for execution will have its scheduled time set as a `setTimeout`

The `startProcessing` will also be called by the `scheduleTask` method. This is because there be no tasks in the schedule, and upon scheduling a task, we trigger the start processing again.

Calling `startProcess` should be idempotent, as in, if the processing is already started, then nothing happens. It would only matter if the `setTimeout` delay should be made smaller because a more recent task has be scheduled.

The queue now would have 2 "queues".

1. Schedule - this is purely time based, by using `IdSortable` as the `TaskId`, this means all tasks have unique scheduling time (up to the maximum amount of ticks the `IdSortable` is capable of). Which means there will always be 1 task that is in front to be executed. Here priority does not matter, it is simply a matter of scheduling time
2. Execution Queue - this is the queue of jobs that are actually pending execution right now, here is where priority actually matters. This can make use of dynamic priority assuming the number of tasks sitting here is not too much. One can just round robin here, or select the task that has sat here the longest. Perhaps a double index sort between initial priority and time of insertion.

If the task is never fulfilled (resolve/rejected), it should stay in the execution queue (which should still be persistent).

Originally in order to "connect" a lazy promise to a task execution, this was done with a callback that I called a "listener".

Now I realised that the deconstructed promise is itself already a set of callbacks to be executed on the task execution.

This means during actual task dispatch, we could just do something like:

ts
taskHandlerExecution(...taskParameters).then(
 resolveTaskP,
 rejectTaskP
).finally(() => {
 this.promises.delete(taskId.toString() as TaskIdString);
});

That is, the promise that comes from executing the task handler gets connected to the deconstructed promise of the task abstraction.

And at the end, the task promise is deleted once the task is done.

This only occurs IF the task promise was first created. If it was lazy promise, it may never get created in which, and if so, nothing is there to observe/await the task execution. That's fine as the task's side effects continues to be done.

So now I have:

* `Queue` - this is an _encapsulated_ dependency of `Scheduler`
* `Scheduler` - this is the main access to the tasks domain

During the `PolykeyAgent` start process, we expect that the `Scheduler` is going to be a *required* dependency of other relevant domains.

ts
const scheduler = new Scheduler();
await Discovery.createDiscovery({ scheduler });
await NodeGraph.createNodeGraph({ scheduler });
await scheduler.start();

Why not use `await Scheduler.createScheduler();`? This is because, this would require us to inject `handlers` from the very beginning, and these handlers are only known by the other domains. Which results in a circular dependency.

Here we are directly constructing the `Scheduler` and using it like a `StartStop` system.

However it is actually CDSS, as there is a `destroy` method too that removes all the persisted state.

One of the issues with this is that certain methods must be possible by the time it is constructed, but not necessarily asynchronously started...

But at the same time asynchronous start is necessary to do any async setup such as creating the database levels... etc.

So now I'm thinking that `start` and `stop` is still used, and thus `createScheduler` is used, but when `Scheduler.start` just doesn't actually start the processing of tasks, necessitating one to call `scheduler.startProcessing()`.

But then it's not going to be symmetric if the `Scheduler.stop` does call `stopProcessing` but `start` doesn't call `startProcessing`.

The alternative is that we use a callback/hook abstraction similar to the `EventBus`. So now instead function hooks is registered in the other domains first. These end up calling the `scheduler` system.

Another alternative is that `Scheduler` is only `StartStop` instead, but again this isn't nice, if there needs to be asynchronous creation routines.

I've added a `delay` boolean to `Scheduler.start` in order to not start the processing. That way users can start scheduling with `Scheduler.startProcessing()` manually afterwards.

By default the `delay` is `false`, so that by default the processing does already start.

This means in the `PolykeyAgent.createPolykeyAgent`, we should instead see something like:

ts
const scheduler = await Scheduler.createScheduler({ delay: true });
await Discovery.createDiscovery({ scheduler });
await NodeGraph.createNodeGraph({ scheduler });
await scheduler.startProcessing();

When stopping the processing this doesn't actually stop the execution of any tasks, it just stops the processing of the scheduler.

The scheduler doesn't _execute_ the tasks. It _dispatches_ to the queue. The queue _assigns_ tasks to workers, workers is what _executes_ the task. At the same time, the workers may also _pull_ tasks from the queue when they are idle.

The `Queue` will need to work with threadsjs queue too: URL I'm not sure yet if this means our `WorkerManager` will need to be changed to work with the `Queue`, since I don't really want there to be 2 queues. Maybe `Queue` is for managing the queue persistence, while embedding the `WorkerManager` in-memory queue that is one to one for each task that is persisted.

Alternatively we actually don't use `WorkerManager` pool, and instead manage our own "pool" directly.

I haven't completed the full design of `Task` class. But I suspect it needs to be similar to lazy promise here: URL and even threadsjs representation uses a `then` method to allow `await` to work on their objects. Their type is:

ts
/**
 * Task that has been `pool.queued()`-ed.
 */
export interface QueuedTask {
 /** @private */
 id: number;
 /** @private */
 run: TaskRunFunction ;
 /**
 * Queued tasks can be cancelled until the pool starts running them on a worker thread.
 */
 cancel(): void;
 /**
 * `QueuedTask` is thenable, so you can `await` it.
 * Resolves when the task has successfully been executed. Rejects if the task fails.
 */
 then: Promise ["then"];
}

If `Task` is in fact a `class Task extends Promise`, it would have properties that would be enumerable, and properties that are not. We may need to specify this explicitly: URL

Alternative is to form an a plain object like threadsjs does instead of using classes.

Note that since `TaskId` is a `IdSortable`, it's strictly monotonic due to our storing of the last task ID...

But this assumes the last Task ID is always stored, and we are intending on deleting tasks off the schedule once completed. I wasn't thinking keeping historical tasks are useful (except for maybe debugging? Although it seems like it would be dropped in production, and logging/tracing systems should be maintaining the audit log).

This means the last task ID may be undefined. So we would store the last Task ID regardless of whether there are any tasks left in the scheduler.

Furthermore, when the clock is shifted backwards, the time will be incremented by 1 until it is greater than the last time. The 1 is the smallest unit of precision, in which case this would be 1 millisecond.

Afterwards, it will be strictly monotonic ID but have a weakly monotonic timestamp up to 4096 IDs per millisecond. After 4096 it would roll over.

The expectation is that it's not possible to generate more than 4096 IDs in a millisecond, so by that time, the time must have increased by at least 1 millisecond.

Anyway this means we need to store `lastTaskId` into the `Scheduler` under level separate from the `Scheduler/tasks` level.

Benchmark in js-workers shows that the overhead to call the workers takes about 1.16 to 1.5ms.

A CPU intensive task should be greater than that time to be worth sending to the worker.

However most scheduling work seems it might not actually be CPU intensive. Like NodeGraph and Discovery is mostly IO. I suppose discovery may have have CPU work to pattern match the data to find the right data on the pages it loads, but this should be dominated by the time spent on IO.

Furthermore sending it to a worker can introduce locking problems. The async locks do not work across the worker threads, they only work within the same event loop context. They are not thread-safe nor process-safe.

This should mean that we should not directly integrate `WorkerManager` into the `Queue`, instead individual domains may have their handlers directly pass work to the `WorkerManager`. The `Queue` does not decide this since it does not know the nature of the task. The domain that registers the handlers can decide the nature of the task. So they can execute within the main thread, or send it off to a web worker and await for it.

This means naturally the `Queue` can have either 1 as a concurrency limit or `0` to indicate unbound concurrent limit. With an unbound concurrency limit, it just immediately proceeds to execute everything that is due for execution.

Priority only comes into play with a concurrency limit so that things get put into priority order. Otherwise all tasks will be asynchronous and immediately executed.

The worker's concurrency/parallel limit is not a concern of the `Queue` then.

Once we have the tasks system, all other domains should not have any kind of background processing implemented, they should delegate ALL of that functionality into the tasks system.

The task management is ready. However integration into discovery and nodes domains is being done in #445.

Priority management is static, we won't bother with dynamic priority in URL before we see it be a problem.

Issue description here is still relevant to #445, since it contains notes on how best to refactor the discovery system.


Question: spire-agent gets OOMKilled after pod restart?

Yes, @Rishikesh01. That is the correct package for the Kubernetes workload attestor code.

spire-agent gets OOMKilled after pod restart
* **Version**: up to 1.8.6
* **Platform**: Linux 6.5.0-26-generic 26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64 GNU/Linux
* **Subsystem**: agent

When I delete a spire-agent pod in the cluster, it becomes unstable, the container gets `OOMKilled` a couple of times before becoming stable.

These are the resource settings currently:

 Limits:
 cpu: 1
 memory: 570Mi
 Requests:
 cpu: 800m
 memory: 512Mi

There is only a warning about a container ID that is not found at attestation then the agent exits with reason `OOMKilled`.

time="2024-04-11T13:40:53Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
time="2024-04-11T13:40:53Z" level=info msg="Starting agent with data directory: "/run/spire/temp""
time="2024-04-11T13:40:53Z" level=warning msg="Agent is now configured to accept remote network connections for Prometheus stats collection. Please ensure access to this port is tightly controlled" subsystem_name=telemetry
time="2024-04-11T13:40:53Z" level=info msg="Plugin loaded" external=false plugin_name=k8s_psat plugin_type=NodeAttestor subsystem_name=catalog
time="2024-04-11T13:40:53Z" level=info msg="Plugin loaded" external=false plugin_name=memory plugin_type=KeyManager subsystem_name=catalog
time="2024-04-11T13:40:53Z" level=info msg="Plugin loaded" external=false plugin_name=k8s plugin_type=WorkloadAttestor subsystem_name=catalog
time="2024-04-11T13:40:53Z" level=info msg="Bundle loaded" subsystem_name=attestor trust_domain_id="spiffe://infra"
time="2024-04-11T13:40:53Z" level=info msg="SVID is not found. Starting node attestation" subsystem_name=attestor trust_domain_id="spiffe://infra"
time="2024-04-11T13:40:53Z" level=info msg="Node attestation was successful" rettestable=true spiffe_id="spiffe://infra/spire/agent/k8s_psat/infra-cluster/890be4ad-1618-4379-9bc9-c54bb55223d5" subsystem_name=attestor trust_domain_id="spiffe://infra"
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=28908b2f-d02b-4269-8ecf-de78d350bf5d spiffe_id="spiffe://infra/ns/infra/pod/nsmgr-72bjt" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=deaeb5c6-ab93-4851-88c5-e13497486f09 spiffe_id="spiffe://infra/ns/infra/pod/forwarder-vpp-xgbgq" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=610ecd0f-e2c9-4e38-922d-9e325a3dd6cb spiffe_id="spiffe://infra/ns/cndsc3/pod/proxy-vpn1-6vkn4" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=0415f6a1-240c-49b3-8215-7e337bcead79 spiffe_id="spiffe://infra/ns/cndsc3/pod/proxy-vpn2-pvb7p" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=0d2837cb-9aa8-4189-b2d7-c3ddeb4ee587 spiffe_id="spiffe://infra/ns/cndsc3/pod/stateless-lb-frontend-attr-vpn2-5444c987npdbt" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=cfdb2a8a-e340-4cef-8d3f-8944af4e4758 spiffe_id="spiffe://infra/ns/cndsc3/pod/fdr-fdcb9f8f6-tqqcj" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Renewing X509-SVID" entry_id=14d49a0a-2b58-4a7c-a0de-551df85d2fc1 spiffe_id="spiffe://infra/ns/infra/pod/registry-k8s-5cc46b8bbf-bk6wc" subsystem_name=manager
time="2024-04-11T13:40:53Z" level=info msg="Starting Workload and SDS APIs" address=/run/spire/sockets/agent.sock network=unix subsystem_name=endpoints
time="2024-04-11T13:40:53Z" level=warning msg="Container id not found" attempt=1 container_id=0c2c4b591c6e68635a6cfff55c91a90244bfb85c2091d0f43874785ff39789a7 external=false plugin_name=k8s plugin_type=WorkloadAttestor pod_uid=1f39d257-6173-4a4e-ac25-3706dd13db63 retry_interval=500ms subsystem_name=catalog
2024-04-11T13:40:58.743 Agent exit code: 137

As it can be seen on this graph, the average memory consumption is always below 200Mi but after restart there is a spike that causes the container restart:
spire-agent-oomkilled

Can you please help to understand if this behavior is normal? Is there any other way than further increasing the memory resource limit to make it stable?

I think this could be related to the fix introduced in #4231. The valyala/fastjson library has some pretty poor memory usage characteristics:

If there is a spike to the number of pods running on the kubelet, the pods response might be quite large. On a 64-bit platform, the size of fastjson.Value is 80 bytes. Even if we assume a 500KiB response, that is 40MiB. This is a per-attestation cost (we don't share the kubelet output, yet).

Further, fastjson has an outstanding bug that causes memory to be held onto a little longer, meaning that if you are undergoing many attestations at once, the GC might not be able to release memory fast enough. There is a PR open but fastjson may not be actively maintained ( URL Hard to know if/when this PR would land.

We've considered moving to another, more frequently maintained library...

We found that even if Spire v1.8.7 is much better than the older releases spire-agent got OOMKilled after a while. So, the spike is still there. Thus, I made a patched version with the parser from valyala/fastjson#101 and tested it. It seems to be working without issues. So, unfortunately it is not enough to upgrade to Spire v1.8.7 or later release.

That's what I suspected. Unfortunately, fastjson seems to no longer be actively maintained. It's probably to the benefit of the project to move to a different json parsing library that is actively maintained.

I've opened #5109 and #5111 to track potential mitigations. I believe #5109 should be done no matter what, considering valyala/json is not actively maintained. If that isn't enough, we could consider #5111, though it is more complicated.

Consider caching the Kubelet response
The Kubernetes Workload Attestor queries the Kubelet for pod information on each attestation attempt. Kubelet responses can be somewhat large (e.g. 1MiB). When many processes are concurrently under attestation, the large response can cause a spike in SPIRE agent memory usage.

This issue is to explore if and how we cache the kubelet response to share across attestation attempts.

Some considerations:
1. How long is this cached for?
2. What events would actively invalidate the cache ahead of expiration? (e.g. expected pod/container not found in response)
3. Each attestation invocation is looking for a different pod/container. Do we cache just the kubelet response and then use targeted parsing of the response to cherry-pick the specific pod/container (what we do today)? Or do we parse the whole response and then share the parsed version (maybe reduces load due to re-parsing)?

Any decisions about the considerations above should ideally be backed by benchmarking to understand the tradeoffs.

Hi @azdagron I am interested in helping out here. I am looking around to find code base where this caching needs to be done and I have found this `/pkg/agent/plugin/workloadattestor/k8s` package I hope this is where I should start looking.

Also about caching part you plan it to be in-memory thing ?


Question: Sokol-shdc segfaults with glsl:100 because of variable shadowing?

I just implemented a fix to communicate the SPIRVCross error message instead of just crashing in branch `storage-buffer`. In case of your example shader the error message isn't very helpful, but better than nothing I guess:

/Users/floh/projects/sokol-tools/test/issue_86.glsl:0:0: error: SPIRVCross exception: Access chains that result in an array can not be flattened

Unfortunately I don't get line information back from SPIRVCross, only an error message. Closing this ticket (even though the fix isn't in the master branch yet).

Sokol-shdc segfaults with glsl:100 because of variable shadowing
The shader:

glsl
@module threedee

@vs vs
in vec3 pos_in;
in vec2 uv_in;

out vec3 pos;
out vec2 uv;
out vec4 light_space_fragment_position;

uniform vs_params {
 mat4 model;
 mat4 view;
 mat4 projection;
 mat4 directional_light_space_matrix;
};

void main() {
 pos = pos_in;
 uv = uv_in;

 vec4 frag_pos = view * model * vec4(pos_in, 1.0);
 gl_Position = projection * frag_pos;

 light_space_fragment_position = directional_light_space_matrix * frag_pos;
}
@end

@fs fs
uniform sampler2D tex;
uniform sampler2D shadow_map;

in vec3 pos;
in vec2 uv;
in vec4 light_space_fragment_position;

out vec4 frag_color;

float decodeDepth(vec4 rgba) {
 return dot(rgba, vec4(1.0, 1.0/255.0, 1.0/65025.0, 1.0/16581375.0));
}

float calculate_shadow_factor(sampler2D shadowMap, vec4 light_space_fragment_position) {
 float shadow = 1.0;

 vec3 projected_coords = light_space_fragment_position.xyz / light_space_fragment_position.w;

 if(projected_coords.z > 1.0)
 return shadow;

 projected_coords = projected_coords * 0.5f + 0.5f;

 float current_depth = projected_coords.z;

 vec2 shadow_map_size = textureSize(shadow_map, 0);
 vec2 uv = projected_coords.xy * shadow_map_size;

 shadow = decodeDepth(texture(shadowMap, uv));

 return shadow;
}

void main() {
 vec4 col = texture(tex, uv);
 if(col.a < 0.5)
 {
 discard;
 }
 else
 {
 vec3 light_dir = normalize(vec3(1, -1, 0));

 float shadow_factor = 1.0;//In percentage of light remaining, i.e. 1.0 full-bright, 0.0 fully shadowed
 shadow_factor = calculate_shadow_factor(shadow_map, light_space_fragment_position);

 frag_color = vec4(col.rgb*shadow_factor, 1.0);
 }
}
@end

@program program vs fs

The issue is in shadowing the `uv` variable, causing a segfault

I was running into a similar (but unrelated) thing yesterday. The reason is most likely an internal error reported by SPIRVCross, but the error handling in SPIRVCross in release mode and with C++ exceptions disabled just silently aborts:

 URL

TL;DR: I need to find a way to report such internal compiler errors from SPIRVCross - for instance by enabling and then catching exceptions from out of SPIRVCross (at least as first step, the next problem will be that those messages won't be very useful in many cases).

PS: wrote a SPIRCross ticket here, but independently from whether that's fixed or not I'll need to find a workaround.

 URL

PS: this was the problem I was running into, also looked like a "silent crash", but was actually a SPIRCross error:

 URL

...and this ticket is similar:

 URL

Once branch `storage-buffer` is merged, this type of problems (internal SPIRVCross compiler errors) will produce a proper error message. Interestingly I cannot reproduce the problem anymore with your shader though (I just did some small changes for the new separate texture/sampler stuff). Maybe updating the Khronos dependencies helped. Closing this ticket, even though the fix (better error message) isn't in master yet.

sokol-shdc segfaults with --slang glsl330:hlsl5
sokol-shdc crashes for me with the following invocations:

sokol-shdc.exe --slang glsl330 --format sokol_zig -i src/shaders/texgrid.glsl -o src/shaders/texgrid.glsl.zig
sokol-shdc.exe --slang glsl330:hlsl5 --format sokol_zig -i src/shaders/texgrid.glsl -o src/shaders/texgrid.glsl.zig

The contents of texgrid.glsl are as follows:

#pragma sokol @vs vs
uniform vs_params {
 vec4 mvp[4];
};

in vec4 position;
in vec2 texcoord0;

out vec2 uv;

vec4 xform(vec4 mat[4], vec4 v) {
 return vec4(dot(mat[0], position), dot(mat[1], position), dot(mat[2], position), dot(mat[3], position));
}

void main() {
 gl_Position = xform(mvp, position);
 uv = texcoord0;
}
#pragma sokol @end

#pragma sokol @fs fs
uniform sampler2D tex;

in vec2 uv;
out vec4 frag_color;

void main() {
 frag_color = texture(tex, uv);
}
#pragma sokol @end

#pragma sokol @program texgrid vs fs

I'm not really setup with a good debugger on this host, but looks like some unhandled exception? ->

Starting program: C:\Users\janne\dev\workspace\fips-deploy\sokol-tools\win64-vstudio-release\sokol-shdc.exe --slang glsl330:hlsl5 --format sokol_zig -i src/shaders/texgrid.glsl -o src/shaders/texgrid.glsl.zig
gdb: unknown target exception 0xc0000409 at 0x7ff665c43b7d

I got the crash on Linux using sokol-tools prebuilt binaries and also on Windows when I built sokol-shdc myself.

Here's the stacktrace that I captured in Visual Studio debugger:

sokol-shdc.exe!issue_debug_notification(const wchar_t * const message) Line 28
 at minkernel\crts\ucrt\src\appcrt\internal\report_runtime_error.cpp(28)
sokol-shdc.exe!__acrt_report_runtime_error(const wchar_t * message) Line 154
 at minkernel\crts\ucrt\src\appcrt\internal\report_runtime_error.cpp(154)
sokol-shdc.exe!abort() Line 61
 at minkernel\crts\ucrt\src\appcrt\startup\abort.cpp(61)
sokol-shdc.exe!spirv_cross::report_and_abort(const std::string & msg) Line 58
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_cross_error_handling.hpp(58)
sokol-shdc.exe!spirv_cross::CompilerGLSL::flattened_access_chain(unsigned int base, const unsigned int * indices, unsigned int count, const spirv_cross::SPIRType & target_type, unsigned int offset, unsigned int matrix_stride, unsigned int __formal, bool need_transpose) Line 9103
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(9103)
sokol-shdc.exe!spirv_cross::CompilerGLSL::access_chain(unsigned int base, const unsigned int * indices, unsigned int count, const spirv_cross::SPIRType & target_type, spirv_cross::AccessChainMeta * meta, bool ptr_chain) Line 8992
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(8992)
sokol-shdc.exe!spirv_cross::CompilerGLSL::emit_instruction(const spirv_cross::Instruction & instruction) Line 10159
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(10159)
sokol-shdc.exe!spirv_cross::CompilerGLSL::emit_block_instructions(spirv_cross::SPIRBlock & block) Line 9860
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(9860)
sokol-shdc.exe!spirv_cross::CompilerGLSL::emit_block_chain(spirv_cross::SPIRBlock & block) Line 14704
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(14704)
sokol-shdc.exe!spirv_cross::CompilerGLSL::emit_function(spirv_cross::SPIRFunction & func, const spirv_cross::Bitset & return_flags) Line 13953
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(13953)
sokol-shdc.exe!spirv_cross::CompilerGLSL::compile() Line 677
 at C:\Users\janne\dev\workspace\sokol-tools\ext\SPIRV-Cross\spirv_glsl.cpp(677)
sokol-shdc.exe!shdc::to_glsl(const shdc::spirv_blob_t & blob, int glsl_version, bool is_gles, bool is_vulkan, unsigned int opt_mask, shdc::snippet_t::type_t type) Line 305
 at C:\Users\janne\dev\workspace\sokol-tools\src\shdc\spirvcross.cc(305)
sokol-shdc.exe!shdc::spirvcross_t::translate(const shdc::input_t & inp, const shdc::spirv_t & spirv, shdc::slang_t::type_t slang) Line 470
 at C:\Users\janne\dev\workspace\sokol-tools\src\shdc\spirvcross.cc(470)
sokol-shdc.exe!main(int argc, const char * * argv) Line 61
 at C:\Users\janne\dev\workspace\sokol-tools\src\shdc\main.cc(61)
[External Code]

Perhaps it'd be fixed by upgrading some of the SPIR-V repos?

It's a bit surprising that SPIRVCross is throwing exceptions instead of a more traditional error reporting, I haven't seen that yet.

It's also strange that such a simple shader generates an error tbh...

I'll try to have a quick look, but may need a couple of days before I can do a proper investigation and fix.

...I vaguely seem to remember problems with array function args though...

Yeah, I remembered right. As a quick workaround, you need to rewrite the xform() function to not take an array arg, e.g.:

glsl
vec4 xform(vec4 mx, vec4 my, vec4 mz, vec4 mw, vec4 v) {
 return vec4(dot(mx, position), dot(my, position), dot(mz, position), dot(mw, position));
}

...and call it like this:

glsl
gl_Position = xform(mvp[0], mvp[1], mvp[2], mvp[3], position);

...doesn't help with the underlying problem of course, but hopefully unblocks you for now.

The error kinda makes sense unfortunately (I think I even reported this already to SPIRVCross, need to find the ticket though).

The GL backend flattens each uniform block into a vec4 array, but since GLSL has no array slices, it cannot access a range within this array for the function parameter (it could create a temporary array though, but I guess that's too much complexity for SPIRVCross for such an 'esoteric' feature).

I'll still try to set aside a bit of time 'soon-ish' to investigate the problem, maybe there's at least a way to make the error less painful (e.g. not crashing, and reporting a more helpful error message).

PS: alternatively this also works, keep the xform() function as it is, but create a temporary array at the call site:

glsl
 vec4 mat[4] = { mvp[0], mvp[1], mvp[2], mvp[3] };
 gl_Position = xform(mat, position)

Thanks for the help! I'll workaround it like above.. I'm mainly using HLSL5 where this was not a problem, just noticed this while I tried to run my code on Linux and OpenGL.

It's probably fine to have a catch all exception handler somewhere in your CLI, then report the exception text and exit. Right now shdc crashes and it's kind of hard to know what happened.

Funny how my quest to go full row major snowballed into even problems like this.. :D

FWIW you also get a segfault when using uint and glsl100 (which doesn't support uint, as expected.) Something a skosh more user friendly than a segfault might be an ergonomic improvement.

IMHO, because SPIRV-Cross seems to SPIRV_CROSS_THROW sort of zealously for error reporting, it might be worth it to just wire `report_and_abort` up to _always_ print the error message. I'm having trouble thinking of a scenario where a user would want sokol-shdc to abort, but not print the underlying error message.


Question: [iOS] "Original error: Command '/usr/bin/python' not found" appear when use setLocation of a iOS Emulator on MAC Apple Silicon (M1)?

Newer versions are available via appium 2. You can get it via npm as `appium@next`.

[iOS] "Original error: Command '/usr/bin/python' not found" appear when use setLocation of a iOS Emulator on MAC Apple Silicon (M1)
## The problem

When I call setLocation function appium return error "Original error: Command '/usr/bin/python' not found" and break test.
<img width="820" alt="Screen Shot 2022-08-08 at 23 10 18" src=" URL

## Environment

* Appium version (or git revision) that exhibits the issue: **Appium 1.22.3**
* Last Appium version that did not exhibit the issue (if applicable): **N/A**
* Desktop OS/version used to run Appium: **macOS Monterey 12.5 - M1Pro**
* Node.js version (unless using Appium.app|exe): **v18.7.0**
* Npm or Yarn package manager: **npm**
* Mobile platform/version under test: iOS 15.5
* Real device or emulator/simulator: XCode Emulator
* Appium CLI or Appium.app|exe: Appium.app

## Details
The error appear when I hit command:
`driver.setLocation(new Location(0,0,0));`

## Link to Appium logs
Gist

## Code To Reproduce Issue [ Good To Have ]
`driver.setLocation(new Location(0,0,0));`

Perhaps duplicate of URL Please try out latest xcuitest-driver with Appium 2.

The issue has been fixed. Please use the recent driver release (only available as part of Appium2)

Appium stop working with iOS on MacOS 12.3
## The problem

Since Mac OS 12.3 python 2 by path usr/bin/python was replaced by Python 3 with path usr/bin/python3
Directory usr/bin/python doesn't exist anymore and can't be created as a protected system directory.

When trying to run any test on a real iOS device or iOS Simulator I get an error:
"org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: Command '/usr/bin/python' not found. Is it installed?"

Any installation of Python2 and adding it to any System path didn't fix a problem.

Looks like WebDriver is looking for usr/bin/python directory only ignoring any System variables and installed python2

Issue reproduced on any call of embedded selenium-remote-driver (3.141.59)
For example: getDriver().setLocation(53.3453, -1.4532, 0);

## Environment

* Appium version (or git revision) that exhibits the issue: GUI appium version 1.22.3-4 + server 1.22.2 and appium console server 1.22.0
* Last Appium version that did not exhibit the issue (if applicable): no one
* Driver version that exhibits the issue (only for Appium2): all (Mac OS version depends)
* Last driver version that did not exhibit the issue (if applicable, only for Appium2): no one
* Desktop OS/version used to run Appium: Mac OS 12.3 (Doesn't reproduce on Mac OS 12.2 and older)
* Node.js version (unless using Appium.app|exe):
* Package manager name and version (we only officially support NPM):
* Mobile platform/version under test: iOS any version
* Real device or emulator/simulator: real device and simulator
* Appium CLI or Appium.app|exe: io.appium java-client 8.1.0
* Java 8

## Details

If necessary, describe the problem you have been experiencing in more detail.

## Link to Appium logs
Intelij IDEA log
 URL

Appium console server log
 URL

Do I need to update Appium library for that ? as i am facing exact issue in mac while setting location
getDriver().setLocation()

I updated appium using `npm install appium -g` but still getting same error , I think appium did not get updated.

Thanks a lot for quick response !! Updated but seems lot of change is required for appium 2.0, had to revert that for now. Any other workaround plz for version 1.22.3?

No, it is impossible to resolve. Also setLocation() it is just an example but not only one method which crashing run by python path error

Unfortunately we don't provide patches for older Appium versions. Closed as fixed

Currently the most recent xcuitest driver version is 4.4.5. It is only available as part of Appium2. Appium1 is is legacy and not maintained anymore

@KazuCocoa When I try set location on Appium2 this issue appear, do you have any advice?

Screen Shot 2022-08-18 at 16 47 13

It looks like Apple has excluded pyobjc from the standard python deployment
We need to avoid python usage in general for such scenarios

Thank you @mykola-mokhnach setLocation function is working well on MacOS latest version (12.5.1)


Question: Compressed, chunked response inhibits connection reuse?

When we return the Stream to the pool, we can check that the `inner` `BufReader`'s buffer is empty.

Compressed, chunked response inhibits connection reuse
rust
env_logger::init();
let agent: Agent = ureq::AgentBuilder::new().build();
for _ in 0..2 {
 let resp = agent.get(" URL
 println!("Connection: {}", resp.header("connection").unwrap());
 std::io::copy(&mut resp.into_reader(), &mut io::sink()).unwrap();
}

results in (RUST_LOG=trace)

[2022-10-17T01:02:37Z DEBUG ureq::stream] connecting to www.jammersreviews.com:443 at 172.67.140.111:443
[2022-10-17T01:02:37Z DEBUG rustls::client::hs] No cached session for DnsName(DnsName(DnsName("www.jammersreviews.com")))
[...]
[2022-10-17T01:02:37Z DEBUG ureq::stream] created stream: Stream(RustlsStream)
[2022-10-17T01:02:37Z DEBUG ureq::unit] sending request GET URL
[2022-10-17T01:02:37Z DEBUG ureq::unit] writing prelude: GET /st-tos/s1/dagger.php HTTP/1.1
 Host: www.jammersreviews.com
 User-Agent: ureq/2.5.0
 Accept: */*
 accept-encoding: gzip
[2022-10-17T01:02:37Z DEBUG rustls::client::tls13] Ticket saved
[2022-10-17T01:02:37Z DEBUG rustls::client::tls13] Ticket saved
[2022-10-17T01:02:37Z DEBUG ureq::response] Chunked body in response
[2022-10-17T01:02:37Z DEBUG ureq::unit] response 200 to GET URL
Connection: keep-alive
[2022-10-17T01:02:37Z DEBUG ureq::stream] dropping stream: Stream(RustlsStream)
[2022-10-17T01:02:37Z DEBUG ureq::stream] connecting to www.jammersreviews.com:443 at 172.67.140.111:443
[2022-10-17T01:02:37Z DEBUG rustls::client::hs] Resuming session
[...]
[2022-10-17T01:02:37Z DEBUG ureq::stream] created stream: Stream(RustlsStream)
[2022-10-17T01:02:37Z DEBUG ureq::unit] sending request GET URL
[2022-10-17T01:02:37Z DEBUG ureq::unit] writing prelude: GET /st-tos/s1/dagger.php HTTP/1.1
 Host: www.jammersreviews.com
 User-Agent: ureq/2.5.0
 Accept: */*
 accept-encoding: gzip
[2022-10-17T01:02:37Z DEBUG rustls::client::tls13] Ticket saved
[2022-10-17T01:02:37Z DEBUG rustls::client::tls13] Ticket saved
[2022-10-17T01:02:37Z DEBUG ureq::response] Chunked body in response
[2022-10-17T01:02:37Z DEBUG ureq::unit] response 200 to GET URL
Connection: keep-alive
[2022-10-17T01:02:37Z DEBUG ureq::stream] dropping stream: Stream(RustlsStream)

As can be seen, though TLS does session resumption, the TCP stream gets closed and reopened between requests. I believe this is caused by the fact that the `GzDecoder` knows when to expect the end of the gzip stream, never tries to read beyond it, and therefore never makes the read that causes the `ChunkDecoder` to give it Ok(0), which is what the `PoolReturnRead` looks for to know when to return the `Stream` to its pool. I was able to get it to work properly by using a `MultiGzDecoder` instead of a plain `GzDecoder`, at least in my private, heavily-modified version of ureq. But I don't think any such thing is available for Brotli.

@OrIdow6 Thanks for reporting this! I've written a test case in #550 illustrating the problem. I don't have a solution yet, but it probably means the `Compression` wrapping of the `PoolReturnRead` needs to inform when it reaches the end.

@OrIdow6 also thanks for taking the time to write such a detailed report. I suspect the MultiGzDecoder only coincidentally fixes this because it will try additional reads after the first gzip stream is done.

In terms of how to fix this: the main problem is that the chunked encoding may have a few more bytes that it hasn't read. Specifically the `0\r\n\r\n` that terminates the response. It's not enough for the `GzDecoder` to say its done; if there are any unread bytes left in the response stream, we cant return it to the pool.

We also don't want to do any syscall-invoking reads in a `Drop` impl, since that could produce surprising hangs.

Proposal: fork chunked_transfer and make it aware of BufRead. When a read is finished, peek at the next 5 bytes in the buffer (if available). If those bytes are `0\r\n\r\n` consume them and set an internal piece of state saying "this stream is done." Expose that state with a `done()` method that can be forwarded by ureq's internal `Done` trait. Implement `Done` for `chunked_transfer::Decoder`. I think we don't actually need to touch the `Compression` layer at all, since all we care about is "did we read all the bytes from the chunked encoding yet."

Note this approach is not perfect: it's possible we could land right on a chunk boundary, with no bytes left in the buffer, but _if_ we read more from the network, the next bytes we would read would be `0\r\n\r\n`. In that case we would fail to recognize we were almost at the end, and would therefore fail to pool the connection. That's probably okay.

Another approach, which would be more perfect: if `chunked_transfer::Decoder` receives a read that ends on a chunk boundary, and there are no more bytes left in the buffer, then it should try to fill the buffer and peek at the next five bytes before returning.

@jsha I pushed a commit URL which potentially fixes the problem without forking anything.

Might be something I haven't thought about, but I think it should be ok.

I did some more thinking and reading about this. What should we do if there is extra garbage after a gzip stream ends? I tested and both curl and Go's net/http generate an error.

According to URL gzip is defined in RFC 1952. In section 2.2:

The GzDecoder docs don't really make it clear, but that decoder only decodes a _single member_. The gzip specification (and therefore HTTP) specify a sequence of members. That's what's implemented by MultiGzDecoder. In other words, MultiGzDecoder is what you want if you actually want to read gzip response bodies, even though its documentation makes it sound like something esoteric.

That plays nicely with our intuition that extra garbage after a gzip member should cause an error. Once you've read a full member, the next thing you read must be either EOF or another fully-valid gzip member. So I take back what I said earlier: it's not really a coincidence that MultiGzDecoder repools, because it will keep reading to EOF.

So, I propose we switch to MultiGzDecoder, which I think should solve this problem in a simple way.

By the way, Brotli doesn't have the same convenient property, where streams happen to be terminated by end-of-file. AFAICT the Brotli decoder behaves similarly to how the `GzDecoder` does, and only consumes the compressed part of its input, possibly leaving some extra garbage on the end. So while `MultiGzDecoder` neatly solves this problem for gzip, we still need to solve it for Brotli.

I think we should error if there is extra garbage. One way to do that: wrap `BrotliDecoder` in a `PreciseReader`. The `PreciseReader`, after getting an `Ok(0)` from the `BrotliDecoder`, calls `into_inner()` on it and does a one-byte read on the underlying reader. If that one-byte read produces anything other the `Ok(0)` that means there's extra garbage and `PreciseReader` should return an error (which leads to not pooling the connection).

Some reasons to error on extra garbage:

 - Would prefer not to silently ignore bad input
 - It avoids an obligation to read arbitrarily many bytes during a single read call
 - curl does it :-D

@jsha great thoughts!

I've pushed a revision of #550 that simplifies the impl somewhat and throws an error if we read anything but 0.

This is fixed (for the gzip case) by switching to MultiGzDecoder in #560! 🎉

For the Brotli case, this will require a little more work. See #559; we'll track the work over there.

Redesigning PoolReturnRead
PoolReturnRead is important because we need to be able to hand something back to the user that (a) implements Read and (b) returns the Stream to the Pool as soon as it is correct to do so. But it relies on a property of the Read interface that is not actually guaranteed and in fact is often not true. It relies on upstream Readers always reading until they get to `Ok(0)`. We tried to fix this with the `Done` trait, for the LimitedRead case. But it's come up again in URL and URL which makes me think we are using the wrong abstraction.

PoolReturnRead only ever wraps two types: `LimitedRead` and `ChunkDecoder`. We can move the "return to pool" logic down a level and make those two types responsible for returning their Stream when it is viable. For instance, `LimitedRead` could have an `Option `, and so long as it has bytes left to read and that `Option` is `Some`, it would read from the inner reader. If any read would leave zero bytes left, it would `take()` the Option and return it to the pool before returning from `read()`. For `ChunkedDecoder` we would need a `BufRead` as the inner reader, so that it could look ahead and see if the next bytes are the end-of-stream indicator, and do the same thing.

The end result of these would be that in almost all cases, the stream would be returned to the pool on the second-to-last read (the one returning `Ok(n)`), not the last read (returning `Ok(0)`). And if the user never calls a read that returns `Ok(0)`, we don't have to care.

Sounds good overall.

Doesn't this mean that we could have garbage data after the body? How do we "guarantee" the `Stream` is correct for the next pool reuse?


Question: Signal UI does not show on Debian Forky unless run with "--disable-gpu"?

Since this issue was opened, we started using `curve25519-n`, a native module. This should improve the performance of our curve25519 code by a lot.

Given that (and the fact that this issue has been inactive for over 3 years), I'm going to close. Happy to reopen if that's wrong.

Signal UI does not show on Debian Forky unless run with "--disable-gpu"
### Using a supported version?

- [x] I have searched searched open and closed issues for duplicates.
- [x] I am using Signal-Desktop as provided by the Signal team, not a 3rd-party package.

### Overall summary

Following me updating all my packages in apt yesterday, Signal's UI stopped working, the app would appear in my panel, and I'd get notifications, but no UI would show. Running signal-desktop with the --disable-gpu flag allows it to launch and work again. When running signal-desktop from the terminal, I can see a bunch of GPU errors. Since I can't get a debug log from the app directly due to it not launching, I've attached that terminal log to the debug log section

### Steps to reproduce

1. Run Debian Forky (with Wayland)
2. Install signal-desktop from Signal's official apt repo
3. Launch signal-desktop
4. UI fails to load

### Expected result

Expected result is for the UI to show.

### Actual result

Current result is the UI does not show at all.

### Screenshots

_No response_

### Signal version

7.78.0

### Operating system

Debian Forky with KDE Plasma 6.5.1 (Wayland)

### Version of Signal on your phone

_No response_

### Link to debug log

NODE_ENV production
NODE_CONFIG_DIR /opt/Signal/resources/app.asar/config
NODE_CONFIG {}
ALLOW_CONFIG_MUTATIONS undefined
HOSTNAME debian
NODE_APP_INSTANCE undefined
SUPPRESS_NO_CONFIG_WARNING undefined
SIGNAL_ENABLE_HTTP undefined
userData: /home/ivy/.config/Signal
(node:16128) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
(Use `signal-desktop --trace-deprecation ...` to show where the warning was created)
[16403:1107/160301.991707:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16403:1107/160301.991816:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16403:1107/160301.991866:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16403:1107/160301.991926:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16403:1107/160301.991968:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16403:1107/160301.992016:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16403:1107/160301.992077:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16403:1107/160301.992125:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16403:1107/160301.992203:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16403:1107/160301.992308:ERROR:gpu/command_buffer/service/shared_image/shared_image_manager.cc:389] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[16403:1107/160301.992388:ERROR:components/viz/service/gl/exit_code.cc:13] Restarting GPU process due to unrecoverable error. Context was lost.
[16128:1107/160301.999767:ERROR:content/browser/gpu/gpu_process_host.cc:964] GPU process exited unexpectedly: exit_code=8704
[16460:1107/160302.077249:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16460:1107/160302.077328:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16460:1107/160302.077377:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16460:1107/160302.077417:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16460:1107/160302.077453:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16460:1107/160302.077485:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16460:1107/160302.077530:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16460:1107/160302.077563:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16460:1107/160302.077593:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16460:1107/160302.077638:ERROR:gpu/command_buffer/service/shared_image/shared_image_manager.cc:389] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[16460:1107/160302.077689:ERROR:components/viz/service/gl/exit_code.cc:13] Restarting GPU process due to unrecoverable error. Context was lost.
[16128:1107/160302.084797:ERROR:content/browser/gpu/gpu_process_host.cc:964] GPU process exited unexpectedly: exit_code=8704
[16505:1107/160302.166760:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16505:1107/160302.166840:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16505:1107/160302.166882:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16505:1107/160302.166925:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16505:1107/160302.166963:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16505:1107/160302.167005:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16505:1107/160302.167047:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=SCANOUT
[16505:1107/160302.167084:ERROR:ui/ozone/platform/wayland/gpu/gbm_pixmap_wayland.cc:82] Cannot create bo with format= RGBA_8888 and usage=GPU_READ
[16505:1107/160302.167118:ERROR:gpu/command_buffer/service/shared_image/shared_image_factory.cc:931] CreateSharedImage: could not create backing.
[16505:1107/160302.167180:ERROR:gpu/command_buffer/service/shared_image/shared_image_manager.cc:389] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
[16505:1107/160302.167243:ERROR:components/viz/service/gl/exit_code.cc:13] Restarting GPU process due to unrecoverable error. Context was lost.
[16128:1107/160302.176041:ERROR:content/browser/gpu/gpu_process_host.cc:964] GPU process exited unexpectedly: exit_code=8704

cannot replicate on 3x Debian unstable (forky) systems, 2 with Intel iGPUs and 1 with AMD Radeon RX 6600, all with KDE Plasma 6.5.2

Simmilar/same error here, signal desktop GUI wont show up, but alert for new message will show up. Running with `--disable-gpu` works.

**Signal:**
oficial repository, version 7.78.0

**GPU:**
OpenGL renderer string: AMD Radeon RX 6900 XT

**OS info:**
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
VERSION_CODENAME=noble

6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC

$XDG_SESSION_TYPE=wayland

**output of `signal-desktop --trace-deprecation`:**

NODE_ENV production
NODE_CONFIG_DIR /opt/Signal/resources/app.asar/config
NODE_CONFIG {}
ALLOW_CONFIG_MUTATIONS undefined
HOSTNAME ai-overlord
NODE_APP_INSTANCE undefined
SUPPRESS_NO_CONFIG_WARNING undefined
SIGNAL_ENABLE_HTTP undefined
userData: /home/ai-overlord/.config/Signal
(node:5982) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
 at asarStatsToFsStats (node:electron/js2c/node_init:2:1796)
 at t.lstat (node:electron/js2c/node_init:2:5284)
 at LOOP (node:fs:2937:15)
 at process.processTicksAndRejections (node:internal/process/task_queues:85:11)
[6028:1108/183801.815645:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.815771:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.815824:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.815866:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.815903:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.815946:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6028:1108/183801.844463:ERROR:components/viz/service/main/viz_main_impl.cc:189] Exiting GPU process due to errors during initialization
[6068:1108/183801.973868:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183801.973967:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183801.974029:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183801.974073:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183801.974113:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183801.974157:ERROR:ui/gl/angle_platform_impl.cc:42] DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
ERR: DisplayEGL.cpp:666 (generateConfigs): RGBA(32,32,32,32) not handled
[6068:1108/183802.001775:ERROR:components/viz/service/main/viz_main_impl.cc:189] Exiting GPU process due to errors during initialization

Good chance this is: URL

I saw this in both the Mattermost Desktop and Signal Desktop snaps that use later electron versions, and have worked around it to try and detect when there is no hardware acceleration and automatically use `--disable-gpu`.

asm.js performance optimization
Our emscripten-compiled curve25519 implementation is here. It can be compiled locally with `grunt build` and build details (compiler flags, sources, etc...) live in `Gruntfile.js`.

It works but is nowhere near native speed.

We should profile and optimize this code.
 URL


Question: Some `cupyx.scipy.signal` APIs fall back to NumPy?

Yes! The new routine was meant for CuPy and other projects to implement `eig` and `eigvals`.

Some `cupyx.scipy.signal` APIs fall back to NumPy
Some `scipy.signal` APIs depend on `numpy.eigvals`, which computes the eigenvalues of a general matrix. However, cuSolver only provides the eigenvalue solver of a symmetric (Hermitian) matrix but does not the one of a general matrix. Therefore, the `cupyx.scipy.signal` counterparts of such APIs are currently implemented by falling back to `numpy.eigvals` or `numpy.roots`, which calls `numpy.eigvals`.

Such APIs include:
- `cupyx.scipy.signal.findfreqs` #7641
- `cupyx.scipy.signal.tf2zpk` #7652
- `cupyx.scipy.signal.ss2tf` #7652

See:
- URL
- URL

Lets add a warning in documentation of these routines regarding device synchronization

Hi @takagi @ev-br @andfoy I could really pick your brains here regarding the general-matrix eigensolver. Why isn't it possible to rewrite the algorithms such that you generate a Hermitian matrix? What's special here? (Context: On the cuSOLVER side, I learned that there's great hesitation in adding the general solver because there's no serious use case. A lot of `eigvals` calls were done, in our opinion, only because people either didn't know `eigvalsh` exists or didn't know their matrix is Hermitian/symmetric. Also, `svd` might help for a lot of use cases ( URL We would love to know what's missing in this understanding.)

Thanks @leofang for letting my head up. As for `numpy.roots`, the roots of a polynomial is found as the eigenvalues of its companion matrix, and the matrix is not symmetric/Hermitian.

 URL

I suppose we need eigenvalues themselves, not just decomposition, so svd may not work.

However, yes, we might use another algorithm for root finding on GPU that does not use eigenvalues.

 URL

I’ll look into other lines that use eigvals...

Implementation of `numpy.linalg.eigvals` equivalent
### Description

NumPy has a `numpy.linalg.eigvals` function, which is often used in real-life programs.
Please provide an equivalent of it for CuPy and name it `cupy.linalg.eigvals`.

### Additional Information

_No response_

CuPy relies on `cuSolver` for implementing these linear algebra routines. However, there are no routines in cuSolver for non-symmetric matrices, so this requires implementing Cuda kernels or custom routines to solve this.

I think we might be able to steal MAGMA's kernels for this purpose 😄

yeah, I njust checked and actually torch uses MAGMA for this!

I'm sorry to disappoint but MAGMA's eigenvalue solver for non-symmetric matrices is really bad. There's no specialized GPU-only path, it uses a mix of LAPACK calls with a tiny bit of CUDA kernels. Eigenvalues are computed using LAPACK.
See lots of `lapack77` calls in URL

Up to about 3k x 3k matrices, it's faster to compute on CPU.

Thanks @IvanYashchuk!!

I was looking into MAGMA code and saw the very same as you, I am just thinking of naively throwing this into a QR solver despite being slow just to avoid CPU sync.

@N0rbert @IvanYashchuk hi

We have discussed this internally, and some people suggest that it is better to use `svd` if possible. And if there is a case in your applications where `svd` can't be used and you really need `eigvals` no matter what.

Thanks

It seems to me that `cuSolver` now has the necessary implementations for non-symmetric matrices:

 URL

If I'm correct, could this request be revisited?

Thanks!

It looks like the latest cusolver has an implementation for non-symmetric matrices. Is it possible to revisit this request? (Looks like it was added in cuSOLVER: Release 12.6 Update 2)
 URL

Hi,

This topic sounds quite interesting to me. Is there anyone already working on this? If not I would like to work on this (and #3255 as well) :)

BR

Remark 3 in the docs for `cusolverDnXgeev` says that it is "a hybrid CPU-GPU algorithm" with "best performance attained with pinned host memory", which leads me to suspect that cuSOLVER uses the same sort of code as MAGMA, with key steps of `geev` (reduction to upper Hessenberg form and the QR algorithm) not being GPU-accelerated. (Also there is no `cusolverDnXggev` to solve generalized non-symmetric/non-Hermitian eigenvalue problems.) Therefore this new routine is unlikely to achieve the kind of orders-of-magnitude performance improvement we get for symmetric/Hermitian eigenproblems. There are a few papers on GPU acceleration of the upper Hessenberg reduction, all many years old, none of which seem to have resulted in widely adopted implementations. This is regrettable, but perhaps not too surprising, as university departments which need to solve large non-symmetric/non-Hermitian eigenproblems tend to have access to compute clusters and need not side-track their scarce resources into developing GPU implementations.

Hi atykhyy,

chiming in here, as I was one of the contributors for the implementation of cusolverDnXgeev.

cusolverDnXgeev is a different implementation and the key steps are GPU accelerated. With eigenvector computation, I would expect a few x speedup compared to Magma.

@codecircuit I installed the most recent CUDA on my Windows machine (not the newest, a Thinkpad P52 with a discrete NVidia GPU Quadro P3200 with Max-Q design, CUDA compute version 6.1), installed cupy from the PR referenced in this thread, and tried the solver on a non-symmetric 10000x10000 matrix with a simple `w,v=cupy.linalg.eig(A)`. I can see that the solver is indeed GPU-accelerated - task manager showed the GPU's CUDA engine chugging at 100% using all 6Gb dedicated GPU RAM - and the solver found the same eigenvalues and eigenvectors of interest to me, but unexpectedly it took 23'40" to complete where scipy's CPU-only solver using default settings (`w,v=scipy.linalg.eig(A)`) took 9'50" on the exact same matrix. Is this too large a matrix size for this amount of GPU RAM, or is my hardware too old, or what's going on?

@atykhyy could it be that you do the computation in FP64? Your graphics card has very poor FP64 performance. Switching to FP32 should speed up the computation but has an effect on the convergence of the algorithm and could result into more iterations. You can see the number of iterations and used precision by enabling cuSOLVER logs by setting environment variable `CUSOLVERDN_LOG_MASK=18`.