Question: some games are missing?

We are aware of the issue. This is related to Windows and MacOS. For now downgrade to 2.2.1, hotfix for that will be released tomorrow.

some games are missing
### Describe the bug

some games like daemon x machina and genshin impact from the EGS are missing from my library in the launcher but present in the official client (I'm on windows)

### Add logs

(02:17:20) INFO: [Legendary]: Location: C:\Users\Shiroe\AppData\Local\Programs\heroic\resources\app.asar.unpacked\build\bin\win32\legendary.exe
(02:17:20) INFO: [Gog]: Location: C:\Users\Shiroe\AppData\Local\Programs\heroic\resources\app.asar.unpacked\build\bin\win32\gogdl.exe
(02:17:24) INFO: [Backend]:
 Heroic Version: 2.2.2 Oden
 Legendary Version: 0.20.25 Our Benefactors
 OS: Microsoft Windows 10 Enterprise KERNEL: 10.0.19044 ARCH: x64
 CPU: Intel Core™ i7-8700K @3.7
 RAM: Total: 34.3 GB Available: 22.5 GB
 GRAPHICS: GPU0: Intel(R) UHD Graphics 630 VRAM: 1024MB DRIVER: GPU1: NVIDIA GeForce RTX 3080 VRAM: 10240MB DRIVER: 511.79

(02:17:24) INFO: [Backend]: User Not Found, removing it from Store
(02:17:24) INFO: [Gog]: Getting data about the user
(02:17:24) WARNING: [Backend]: Protocol already registered.
(02:17:25) INFO: [Gog]: Saved user data to config
(02:17:25) INFO: [Frontend]: Refreshing Library
(02:17:25) INFO: [Gog]: Getting GOG library
(02:17:26) INFO: [Gog]: Number of library pages: 2
(02:17:26) INFO: [Gog]: Getting data for page 2
(02:17:26) INFO: [Gog]: Saved games data
(02:17:26) INFO: [Legendary]: Refreshing library...
(02:17:26) INFO: [Frontend]: No cache found, getting data from legendary...
(02:17:26) INFO: [Legendary]: Refreshing library...
(02:17:26) INFO: [Gog]: Found 0 game(s) to update
(02:17:30) INFO: [Backend]: checking for heroic updates
(02:17:45) INFO: [Legendary]:
(02:17:45) ERROR: [Legendary]: [cli] ERROR: Legendary is missing game metadata, please login (if not already) and use the "status" command to fetch necessary information to set-up syncing.

(02:18:05) INFO: [Backend]: Changing Language to: en
(02:18:14) INFO: [Backend]: Changing Tray icon Color...
(02:22:58) INFO: [Frontend]: Refreshing Library
(02:22:58) INFO: [Gog]: Getting GOG library
(02:22:59) INFO: [Gog]: Number of library pages: 2
(02:22:59) INFO: [Gog]: Getting data for page 2
(02:22:59) INFO: [Gog]: Saved games data
(02:22:59) INFO: [Legendary]: Refreshing library...
(02:22:59) INFO: [Frontend]: No cache found, getting data from legendary...
(02:22:59) INFO: [Legendary]: Refreshing library...

### Steps to reproduce

1. Go to library
2 games are missing

### Expected behavior

the previously mentioned games to show up in the list

### Screenshots

Screenshot_342

### System Information

- OS windows
- Heroic Version 2.2.2 Oden

### Additional information

_No response_

This is a current issue (see #1053). Until this is fixed use v2.2.1.

No I don't think so because, unlike the issue you linked me to, I can see certain games but not others

Because I used the search to show you the game that didn't show up but some others do appear Il 9 mar 2022 08:59, Flávio F Lima ***@***.***> ha scritto:
I cannot see any game on your library.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

Daemon X Machina is also not showing for me in Heroic but also in the main client, also having this issue with Tiny Tina's Wonderlands

After the last update (done a few minutes ago) the issue seems to be solved

@flavioislima I think this can be closed, issue seems to be solved.

No longer seeing Library after upgrading to 2.2.2
### Describe the bug

I just upgraded to 2.2.2 after being prompted to upgrade, and now I cant seem to get anything after logging in to Epic games.

### Add logs

(10:55:30) INFO: [Legendary]: Logging with Legendary...
(10:55:30) INFO: [Legendary]: Logging with Legendary...
(10:55:30) INFO: [Legendary]: stderr: [cli] INFO: Testing existing login data if present...

(10:55:30) INFO: [Legendary]: stderr: [Core] INFO: Trying to re-use existing login session...

(10:55:30) INFO: [Legendary]: stderr: [cli] INFO: Testing existing login data if present...
[Core] INFO: Trying to re-use existing login session...

(10:55:31) INFO: [Legendary]: Logging with Legendary...
(10:55:31) INFO: [Legendary]: stderr: [cli] INFO: Stored credentials are still valid, if you wish to switch to a different account, run "legendary auth --delete" and try again.

(10:55:31) INFO: [Legendary]: stderr: [cli] INFO: Stored credentials are still valid, if you wish to switch to a different account, run "legendary auth --delete" and try again.

(10:55:31) INFO: [Legendary]: finished login
(10:55:31) INFO: [Legendary]: finished login
(10:55:31) INFO: [Legendary]: stderr: [cli] INFO: Testing existing login data if present...

(10:55:31) INFO: [Legendary]: stderr: [Core] INFO: Trying to re-use existing login session...

(10:55:31) INFO: [Legendary]: stderr: [cli] INFO: Stored credentials are still valid, if you wish to switch to a different account, run "legendary auth --delete" and try again.

(10:55:31) INFO: [Legendary]: finished login
(10:55:36) INFO: [Frontend]: Refreshing Library
(10:55:36) INFO: [Legendary]: Refreshing library...
(10:55:36) INFO: [Frontend]: No cache found, getting data from legendary...
(10:55:36) INFO: [Legendary]: Refreshing library...

### Steps to reproduce

1. Go to App
2. Log in
3. No library

### Expected behavior

Go to App
Log in
See Library

### Screenshots

_No response_

### System Information

- Mac OS Monterey
- Heroic 2.2.2

### Additional information

_No response_

Or maybe go to Manage Accounts and click continue. I thought this bug was fixed but maybe it wasn't on Mac.

I had a similar issue, i just had to select epic games under library.

I can't access my library as well. Tried uninstalling and installing again and nothing happenned. I'm using macOS Monterey on Mac Mini M1.

I tried but it is the same after I log in

Sent from my iPhone

On Mar 7, 2022, at 11:03 AM, Flávio F Lima ***@***.***> wrote:

Thats pretty weird. Can you try to Logout and Login Again?

—
Reply to this email directly, view it on GitHub


Question: Allow for custom AgroalDataSourceListener?

I think we should aim at fixing this thing for the next LTS, either by going Yoann's way ^. It's not the first time that we had issues with shutdown ordering so taking a step back and listing all the issues to find a global solution would be awesome.

In the meantime, I wonder if we should include URL in the `quarkus-jdbc-oracle` extension.

I will create a PR with ^ to get the ball rolling.

Allow for custom AgroalDataSourceListener
### Description

I would like to be able to configure a custom AgroalDataSourceListener.

I found the following issue from 2019, but the change only allowed for a custom AgroalPoolInterceptor:
"Allow addition of AgroalDataSourceListener implementations during run-time production of DataSources."
 URL

### Implementation ideas

I would propose to have a similar solution for custom AgroalDataSourceListener as there currently is for AgroalPoolInterceptor.

A custom AgroalPoolInterceptor can be defined by making an ApplicationScoped implementation of the interface.
It is also possible to annotate the interceptor class with the @DataSource qualifier.

The exact same setup could be done for custom AgroalDataSourceListener.
The change would have to be done in io.quarkus.agroal.runtime.DataSources.

private final Instance agroalPoolInterceptors; // existing
private final Instance agroalDataSourceListeners; // to be added

Currently there is only 1 hard-coded listener for logging
This part should be adapted to include the custom agroalDataSourceListeners as well

AgroalDataSource dataSource = new io.agroal.pool.DataSource(agroalConfiguration,
 new AgroalEventLoggingListener(dataSourceName,
 agroalConfiguration.connectionPoolConfiguration()
 .transactionRequirement() == TransactionRequirement.WARN));

Hi. Is this different from URL ?

@barreiro , the person most likely to approve such a feature, had concern with the approach: URL
Do you have ideas to address those concerns?

All valid concerns, I guess it's the responsibility of the developer to know what they are doing.

The AgroalPoolInterceptor that can be used now also gives access to the underlying connection.
Maybe it has less room for errors, but I'd say there are infinite ways to mess up if not careful :)

For context, I need to make sure that connections are rolled back before they are closed.
Oracle seems to perform a commit when a connection is closed.
We have an issue when quarkus is shut down gracefully while there are long-running tasks running.
I created an issue for this specific problem as well: URL

Would it be possible to get an update on this, we currently have to resort to pretty ugly code to have a custom AgroalDataSourceListener so that transaction management for Oracle works properly under Quarkus.
See URL for more details on that

@barreiro
Would it be possible to look at this issue?
We currently have to rely on a dirty workaround to have transaction integrity. This would allow us to have proper code.

Alternatively, Quarkus could have a built-in datasource listener that does the rollback before close. (needed for Oracle)

 @Override
 public void beforeConnectionDestroy(Connection connection) {
 try {
 connection.rollback();
 } catch (SQLException e) {
 // ignore
 }
 }

@segersb Agroal 2.3 adds `onConnectionCreate` / `onConnectionDestroy` methods to `AgroalPoolInterceptor`.

This will support you use case (if Agroal prevents the rollback, you'll need to unwrap the connection and call `rollback` on the raw connection).

This methods run on Agroal housekeeping thread and therefore should not be abused.

@barreiro
Thanks, that would already be a much better solution than what we have now!

Is that something that Quarkus would implement by default?
It doesn't seem right that transactions are not safe by default when using Oracle.

Changed the title to clarify that the solution was not to allow custom `AgroalDataSourceListener` in the end, but this:

Unexpected commit of transaction using Oracle
### Describe the bug

The Oracle JDBC driver performs a commit when a connection is closed without an explicit commit or rollback.
When Quarkus is shut down gracefully, connections are closed properly, and this causes unfinished transactions to be committed.

Consider the following simple rest endpoint.
We insert a record in the database, wait 2 minutes and then throw an exception.

@PermitAll
@Path("/test")
public class TestResource {
 @Inject
 EntityManager entityManager;

 @GET
 @Transactional
 @SneakyThrows
 public Response test() {
 entityManager.createNativeQuery("insert into TEST_TABLE (TEST_COLUMN) values (1)").executeUpdate();

 System.out.println("shut down app now");
 Thread.sleep(120000);

 if (true) {
 throw new RuntimeException("app was not shut down, throwing exception to avoid transaction commit");
 }

 return Response.ok().build();
 }
}

### Expected behavior

I would expect that nothing is inserted in the database.

### Actual behavior

When doing a graceful shutdown during the wait period, the insert is committed.

### How to Reproduce?

See code above for the simple rest endpoint.

To perform a graceful shutdown locally we need to start quarkus-dev via a java command, and not `mvn quarkus:dev`.
Fortunately when I execute quarkus-dev via IntelliJ, the java command is the first line of the output.
This allows us to use CTRL-C, which will do a graceful shutdown.

1. run the java command to start quarkus:dev
2. call the rest endpoint
3. wait for the quarkus console to output "shut down app now"
4. press CTRL-C and wait for the Quarkus to shut down
5. check the database

### Output of `uname -a` or `ver`

_No response_

### Output of `java -version`

_No response_

### GraalVM version (if different from Java)

_No response_

### Quarkus version or git rev

_No response_

### Build tool (ie. output of `mvnw --version` or `gradlew --version`)

_No response_

### Additional information

_No response_

I didn't try this locally yet (have to download an Oracle container on a low-bandwidth connection...), but IIUC the main problem is that the interceptor doesn't trigger a rollback before Agroal gets stopped (and connections get closed).

I wonder if the root cause isn't simply that your thread gets interrupted, leading `Thread.sleep` to throw an `InterruptedException`, which, crucially, isn't a `RuntimeException` and thus doesn't trigger a rollback...

See URL

(And yes I agree it's a crazily dangerous default, but that's the spec.)

Could you try annotating your method with this instead, and tell me if it solves your problem?

java
@Transactional(rollbackOn={Exception.class})

Thanks for your reply!

The result is still the same.
I used Thread.sleep in a rest endpoint here to make a simple example.
We noticed the issue with longer running processes combined with redeployment on Openshift.

I also verified this "commit on close" behavior with basic JDBC.
If I don't explicitly roll back, then the insert is committed when closing the connection.
My guess is that Oracle is the only database provider that behaves like this.

 try {
 Class.forName("oracle.jdbc.driver.OracleDriver");
 conn = DriverManager.getConnection(jdbcUrl, username, password);
 conn.setAutoCommit(false);

 String insertSQL = "INSERT INTO D00SREFTPB01.TEST_TABLE (TEST_COLUMN) VALUES (?)";
 try (PreparedStatement pstmt = conn.prepareStatement(insertSQL)) {
 pstmt.setLong(1, System.currentTimeMillis());
 pstmt.executeUpdate();
 System.out.println("Dummy data inserted into TEST_TABLE");
 }

// conn.rollback();
 } catch (SQLException | ClassNotFoundException e) {
 e.printStackTrace();
 } finally {
 if (conn != null) {
 try {
 conn.close();
 } catch (SQLException e) {
 e.printStackTrace();
 }
 }
 }

That's for sure, and from what I can see it's well-known to Oracle users; found this in particular.

I'm not too worried about Oracle's behavior though, more about the fact that the transaction wasn't rolled back by the interceptor (so, before the connection was closed), especially with `@Transactional(rollbackOn={Exception.class})`.

There must be something wrong on shutdown, maybe Agroal stopping before application threads, I don't know. Needs further investigation.

Note I did fix the issue with a workaround by using a AgroalDataSourceListener.

The code is dirty as there is no out-of-the-box solution to set a custom AgroalDataSourceListener in Quarkus.
I also created a feature request to allow for that ( URL

But in essence I just do a rollback in `beforeConnectionDestroy`
Normally if the transaction was successful, Quarkus should have performed the commit and there would be nothing to roll back.

@ApplicationScoped
public class CustomOracleRollback implements StartupInitializer, AgroalDataSourceListener {
 @Inject
 DataSources dataSources;

 @Override
 @SneakyThrows
 @SuppressWarnings("all")
 public void initialize() {
 DataSource dataSource = (DataSource) dataSources.getDataSource(" ");

 Field connectionPoolField = DataSource.class.getDeclaredField("connectionPool");
 connectionPoolField.setAccessible(true);
 ConnectionPool connectionPool = (ConnectionPool) connectionPoolField.get(dataSource);

 Field listenersField = ConnectionPool.class.getDeclaredField("listeners");
 listenersField.setAccessible(true);
 AgroalDataSourceListener[] listeners = (AgroalDataSourceListener[]) listenersField.get(connectionPool);

 List newListeners = new ArrayList<>(Arrays.asList(listeners));
 newListeners.add(this);

 Field theUnsafeField = Unsafe.class.getDeclaredField("theUnsafe");
 theUnsafeField.setAccessible(true);
 Unsafe unsafe = (Unsafe) theUnsafeField.get(null);

 long offset = unsafe.objectFieldOffset(listenersField);
 AgroalDataSourceListener[] newListenersArray = newListeners.toArray(new AgroalDataSourceListener[0]);
 unsafe.putObject(connectionPool, offset, newListenersArray);
 }

 @Override
 public void beforeConnectionDestroy(Connection connection) {
 try {
 connection.rollback();
 } catch (SQLException e) {
 // ignore
 }
 }
}

Note the issue is only happening on graceful shutdowns.
I'm not sure how this is handled by Quarkus, but my guess is that the interceptor you are referring to never gets executed.
Maybe the executor thread pool is just shut down with a timeout and threads are killed if they take too long to finish.

That's the thing: if threads are killed, the call to `Thread.sleep` should throw an `InterruptedException`, and if you put `@Transactional(rollbackOn={Exception.class})` on your method, that should trigger a rollback.

And all that should happen before the DB connection gets closed, because why on earth would Quarkus stop DB connection pools before killing application threads...

If it doesn't happen that way, there's something wrong...

Another theory would be that `@SneakyThrows` somehow confuses Quarkus and leads to the interceptor misbehaving. While unlikely, I suppose that's possible.

Unfortunately I didn't have time to make progress on this.

I had to work on the Agroal code recently, and noticed this:

 URL

If Vert.x threads are killed in the same way, there's indeed a chance the order is undetermined (vertx threads may be killed before or after agroal connection pools, depending on who knows what)

Maybe killing the applications threads don't cause an explicit rollback of the transaction?

I can't really comment on Quarkus internals, but the root cause of my issue is that Oracle commits on close.
The dirty workaround I have now simply makes sure there is always an explicit rollback before the close.
I believe the custom listener I made is triggered by the `dataSource.close()` in the code you mentioned.

I'm not sure what the best solution is for Quarkus.
The "explicit rollback before close" solution seems pretty robust to me.
But you could argue it's redundant and should be handled by the code behind @Transactional and the shutdown order.

If you used `@Transactional(rollbackOn={Exception.class})` (and you said earlier you did try that), then `InterruptedException` definitely should cause a rollback, at least that's how I understand the spec.
But yes, maybe the bug is there, we'll need to check.

Yeah that... is really a dubious design choice. I guess we could override this behavior by default if there's a setting in the JDBC driver, but then if you had to resort to complicated listeners, I guess there isn't such a setting.

Exactly. Because there might be other, similar problems caused by this (still hypothetical) wrong shutdown order.

HI!

Probably, we also encountered the mentioned unexpected commit.

This commit occurred for us when we tried to shut down Wildfly(27) gracefully. At that time, we noticed that in some cases, only a part of the tables was committed to the Oracle database (we experienced the same on PostgreSQL).

see:
 URL

There exists a property in version 1.5.9.Final ironjacamar jdbc where this can be set, and after that, the cleanup function handles the commit as expected for us (checked the main branch, its also looks good to me):

 URL
(JAVA_OPTS: "-Dironjacamar.jdbc.setautocommitoncleanup=false")

Without this, we also observed that records appeared in the database that shouldn't have.
I think Quarkus also uses similar libraries; perhaps it will help.

Some tickets we started from:
 URL
 URL

Thanks for the input!

I tried looking for some property to set before, but didn't find any.
Maybe the Quarkus devs can work with this.

In sqlplus it's the same behavior and you can turn it off with `set exitcommit off` (it's on by default)
However I was unable to find the same setting on the OJBDC driver.

Hey @segersb , while working on something else I stumbled upon this: URL

Could you confirm that when you talk about graceful shutdowns, you mean it in this specific sense (you set `quarkus.shutdown.timeout` in your app), not in a more general, technology-agnostic sense? Because apparently Quarkus shutdowns are not, by default, graceful.

This probably doesn't change the problem, but would be interesting to know.

Hello @yrodiere

We haven't set anything specific for that parameter, so we just use the default.

We detected the issue when running our Quarkus app on OpenShift.
We have a batch with a long transaction that inserts records in a database.
Of course we expect that either all the records or none are inserted.

When a redeploy is done in OpenShift while this batch is running, we end up having partial data.

When running the standard `mvn quarkus-dev` + CTRL-C locally, we could not replicate this issue.
When pressing CTRL-C the process is terminated and the prompt is shown instantly.

When running the quarkus:dev via java.exe + CTRL-C locally, we were able to replicate the issue.
When pressing CTRL-C it takes a while before the process is terminated and the prompt is shown.

This led us to the conclusion that the shutdown method is somehow involved.
When the issue occurs it seems that the connection is explicitly closed, causing Oracle to commit on close.
When the application is instantly terminated there is no issue since Oracle does no commit when the connection is lost.

Hi, we also have this exact issue. First observed in our Kubernetes-based test environment, where Kubernetes sends a SIGTERM to the Quarkus app, after which a partial transaction got committed to the Oracle DB.

I can reliably reproduce it using Quarkus dev mode and pressing `CTRL+C` or `q` during a large transaction with 100k+ inserts to multiple tables. Then, some thousands of table entries are visible, but not all.

Using the workaround from @segersb above ( URL almost, but not quite, fixes the issue. The call to rollback happens without throwing an exception, but a SINGLE table entry is still inserted anyway.

One thing I've noted is that "Arjuna" also tries to rollback the transaction, but fails because the connection was already closed:

`[co.ar.at.jta] (database-worker-saving-entities-1) ARJUNA016045: attempted rollback of (io.agroal.narayana.LocalXAResource@5158e0b4) failed with exception code XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to transactionRollback local transaction: ORA-17008: Closed connection
`

I'd be very happy if Quarkus could reliably rollback all open transactions BEFORE closing the connections.
We're currently using Quarkus 3.6.8, graceful shutdown timeout not enabled.

Hi @ivu-mawi,

Thanks for your input!

Does your project maybe have multiple datasources?
I'm hoping that might explain why the workaround is not fully working on your end.
It was specifically made for our use case with only the default datasource:
`DataSource dataSource = (DataSource) dataSources.getDataSource(" ");`

If not I'm a bit worried we'll also encounter issues in our projects.

Hi, also thanks for reporting this and the workaround.

I'm afraid we also use a single, default datasource.

The single inserted entry doesn't always occur. But I'm guessing it only doesn't occur when I press `CTRL+C` too quickly before hibernate has decided to actually send statements to the driver or something.

I think my workaround now will be to simply `Thread.sleep` in a method that `@Observes ShutdownEvent` and only returns when all currently active transactions have finished. (We have some code that wraps all relevant transactions, so I'll keep a counter there and sleep until it reaches zero.)

This might mean waiting too long, so Kubernetes will SIGKILL our app. But this leaves no chance to properly close the oracle connections, and even Oracle does a rollback then in my tests, same as `CTRL+C` on Windows as you mentioned.

But I don't know how this would interact with any other ShutdownEvent Observers, if we had any. Would this delay the other ones, preventing their graceful shutdown until SIGKILL happens? Not a good solution in that case.

That's a creative workaround!

Normally you should be able to use @Priority annotation to make sure other shutdown listeners go first.
We used that in the past for startup listeners, so I guess it will also work for shutdown.

SIGKILL always worked for me as well because Oracle only commits when connections are closed properly.
I guess a shutdown listener that sleeps forever could also work, effectively forcing kube/oc to kill the pods.

Another approach might be to combine the Quarkus property `quarkus.shutdown.timeout` and the kube property `terminationGracePeriodSeconds`
`quarkus.shutdown.timeout=30S` would make sure Quarkus does not start closing connections after 30S to be sure
`terminationGracePeriodSeconds: 0` would cause kube to kill the pods

Thanks, `@Priority` works!

I agree with the rest.

I'll stick with my "dynamic" delay, because it allows a normal graceful shutdown if no transactions are running, or very short transactions, which I expect is the much more likely scenario for our app. So the SIGKILL from Kubernetes would only be a rare last resort kind of thing, if someone happens to trigger a restart during one of those rare long transactions.

Still, would be much nicer if Quarkus rolls back before close of course :)

Great it helped!

Interestingly the doc for using `quarkus.shutdown.timeout` says: "Setting this timeout will incur a small performance penalty, as it requires active requests to be tracked." I guess they have a similar code to yours to track the open transactions.

If you still have issues, you could also play with the values of those properties.
`terminationGracePeriodSeconds` can be set to maybe 30 second or so while `quarkus.shutdown.timeout` could be set to a minute.
In a "normal" redeploy Quarkus should be able to shut down within the grace period op kube.

@yrodiere @gsmet
Since I'm not alone having this issue, would it be possible to have a solution in Quarkus?
The fact that we risk having partial transactions committed when using Oracle is not reassuring.
When updating Quarkus we always have doubts that out dirty workaround keeps working properly.

I agree it's a problem, and I agree it should be solved. It's on my todo-list, but that todo-list is long and does not include just Quarkus.

If you want this to be solved quicker, feel free to send a PR or report this through paid support (on RHBQ, ...).

Failing that, you can rely on workarounds (I think a few were mentioned?) and I guess you can hope that the upgrade of Agroal #39072 will change something.

Stumbling across this after days of investigating a swath of lost data that was a consequence of some intermittent state being erroneously committed, I am thankful to have found this discussion and being able to present the workaround originally presented by @segersb in URL with the new capabilities added by URL incorporated to simplify it:

java
/**
 * Oracle has the weird behavior that it performs an implicit commit on normal connection closure (even with auto-commit disabled),
 * which happens on application shutdown. To prevent whatever intermittent state we have during shutdown from being committed,
 * we add an explicit rollback to each connection closure. If the connection has already received a COMMIT, the rollback will not work, which is fine.
 *
 * The code unwraps the {@link Connection} so that we perform the rollback directly on the underlying database connection,
 * and not on e.g. Agroal's {@link ConnectionWrapper} which can prevent the rollback from actual being executed due to some safeguards.
 */
@ApplicationScoped
public class CustomOracleRollback implements AgroalPoolInterceptor {
 @Override
 public void onConnectionDestroy(Connection connection) {
 try {
 connection.unwrap(Connection.class).rollback();
 } catch (SQLException e) {
 Log.tracef(e, "Ignoring exception during rollback on connection close");
 }
 }
}

Hello @Felk

I also did the same on our side, though I did have to add an extra check before doing the rollback.
Not sure you need this, but mentioning it just in case.
We also use flyway on our side, and if I always did the rollback, the migration failed.

The `connection.isClosed()` check was just to avoid exception logging.
The `connection.getAutoCommit()` check was the important one that fixed the flyway migrations.

 @SneakyThrows
 @Override
 public void onConnectionDestroy(Connection connection) {
 if (connection.isClosed() || connection.getAutoCommit()) {
 return;
 }

 OracleConnection oracleConnection = connection.unwrap(OracleConnection.class);
 oracleConnection.rollback();
 }

Good point, thanks! I edited my comment to include those two conditions

This is starting to look like the most likely explanation given what I learned in URL

So a possible fix would be to make sure Agroal shuts down after Vertx... that probably means the Vertx extension needs to use `ShutdownEvent`. We'd need to check if that's already the case; if so, then the problem is elsewhere.

Vert.x should be the last to shut down. Otherwise, many things would not be functional.

We can imagine an SPI indicating when Vert.x has been shut down (it's an async operation so after completion) for the cases where things must happen after (it should be rare and used with a lot of caution)

I see, and yes I understand, as Hibernate Reactive for example would no longer work after vertx shut down. And may not be able to shut down, as shutdown in Hibernate Reactive may imply executing SQL statements...

Could we perhaps have an event after vertx has stopped accepting new incoming requests and finished processing existing requests? That's more or less what we need here: the guarantee that nothing external will attempt to initiate use of datasources.

Though I suppose other components unrelated to vertx (Cron? other servers with more exotic protocols?) could be triggering something similar. We may want a different form of synchronization... a two-step shutdown perhaps?

I created URL for discussion. Thanks for all the good discussions in this issue!


Question: Add Allocs page enabling Jobs -> Allocs -> Tasks flow.

@cshintov this should be fixed in `next` (`go install github.com/robinovitch61/wander@next`) now. Mind giving it a shot when you get the chance and letting me know if there is any other undesirable behavior?

Add Allocs page enabling Jobs -> Allocs -> Tasks flow.
**Is your feature request related to a problem? Please describe.**
I'm running my blockchain nodes on Nomad and many chains are run with many sidecars (tasks) together.
My interaction with them is always focused on a particular alloc and particular task. They're pets and not cattle.

tasks-allocs-merged-view

focused-on-single-alloc

Hence I value the ability to focus on Allocs and then Tasks associated with it. The merged view is distracting and
requires extra work to filter out the noise.

**Describe the solution you'd like**
If I can drill down from Jobs -> Allocs -> Tasks, it would be a great UX.
Peace of mind, less noise.

**Describe alternatives you've considered**
I can use the filter, but it's not as intuitive as the drill down. Besides, it has a nagging bug; after the initial filter
if I move to next filtered row, after the tick, the selection jumps back to the first filtered row. Also it doesn't hide
the unfiltered rows.

But even if the filter works perfectly, it's not as intuitive as the drill down.

**Additional context**
I've played around and implemented the drill down version along with alloc restart in in #126. Would be great to have it in the mainline.

Thanks for opening this for discussion @cshintov . To summarize, wander currently has 2 levels of drill-down: 1) jobs and 2) allocations and tasks combined in one table. This proposal would change that to 3 levels: 1) jobs, 2) allocations and 3) tasks, more similar to the web ui's model.

I'm interested in getting more data before proceeding with this change, as I personally see the 2 level model as an advantage, but am open to shifting to 3 level if there's broader support.

Ah, that's super annoying behavior! Broke it out into a bug issue here: URL

If you run wander with `--filter-with-context=false` or the equivalent env var/config option, it will only show matching rows

If you want to extract out the alloc restart functionality and PR just that against `next` right now in a separate PR, we should be able to get that merged quickly regardless of the outcome of this drill-down discussion!

@coderigo @classabbyamp @qwo @viniciusartur @Antse @sheridanvk @ghthor @yuankaiyu @hoppel @chuckyz @foozmeat @neomantra @bbarg @dbalan @beakman @moyiz ,

You've all contributed to or opened issues in wander before. Assuming you are or were users of it, what do you think of this proposal:

I'm personally a fan of the existing drill down levels, though I haven't run into a case as the OP where it is hindering any of my workflows.

My question to the @cshintov. If you could group/sort by the alloc ID in the combined view, would that improve your workflow? That's an additional sorting mode, where the current sorting mode is based on the task name only.

I too like the current drill-down levels, I don't know that layering it one
extra level adds much from when I've used it as by the time I get to a job
things tend to easily fit on one screen without paginating and it's obvious
which alloc belongs to which task.

On Tue, Feb 6, 2024 at 3:14 AM Will Owens ***@***.***> wrote:

Ya, this could work if combined with `--filter-with-context=false`, **but still more work - more key presses and brain cycles**.

Although I understand why the Web GUI is that way, I find myself jumping a lot between the Alloc/Task windows in the Web GUI. And very often I want to check out the `alloc` files but that's not directly available from the `task` so you have to jump around, etc. But then for an alloc you often need some minor task info and need to pop into the task page.

So I like how `wander` does it currently, but I don't have as many allocs/tasks as OP.

I thus appreciate that one drills down from Allocs into a Task; but I'd still want the current display at the Alloc level.

The power of TUI is that you can get to the where you want very easily -- enter/escape into a Task view is consistent with this. That said, if a filter/sort helps achieve the same, then that might be simpler.

Is there any particular content/UX in a Task pane that would be different than the Alloc-focused pane? In OPs example, they look identical.

This is a great idea. What if we sorted such that the "all tasks for job" page's first rows were the first task in each allocation, eg

Alloc A Task 1
Alloc B Task 1
Alloc C Task 1
Alloc A Task 2
...

And pressing "t" on one of these rows brings you to a "tasks only for this allocation" page. Do you think this is a nice middleground @cshintov ?

I think this is the reason why I want a focused view, and others don't. At small numbers (without many sidecars) it doesn't matter much.

Ya, the above proposal might work, but I don't think it's needed, because without `filter with context` there's already a middle ground.

But if the majority of users are fine with the current view, it's OK.

Besides, I can maintain my own version for my use case; that's the beauty and purpose of open source, isn't it?

| Besides, I can maintain my own version for my use case; that's the beauty and purpose of open source, isn't it?

The other beauty is that the openness allows us to have broader awareness so that we can all be more powerful together. I certainly want large Nomad installations to exist and operate well, even though I don't have a large one myself...

If you do experiment with your idea and make a PR or fork, please note it on this issue =)

Sweet, thanks all for the discussion here. Sounds like `--filter-with-context=false` is a good enough middle ground for now. I'd be happy to add the "t for allocation-specific tasks" implementation here if desired in the future, but will close this issue for now as it seems like no one needs it yet. Anyone feel free to reopen or open a new issue referencing this one if you want to.

@neomantra

I had first raised the PR and then opened the issue 🙂
You can find it here: #126

Glad to hear that you loved Kerala. Seems like you are fond of India. Your profile pic, the handle, etc! 🙏

Search selection jumps back to first result on page reload/refresh
**Describe the bug**
Search selection jumps back to first result on page reload/refresh

**To reproduce**
Run `wander -u 1` in "filter with context" mode on (the default), filter for results, hit `n` to see the next result, and wait. Selection will jump back on page refresh

**Expected behavior**
Filter selection should be preserved after refresh to the best ability (if result disappears on refresh, should be as close as possible to result)

**Additional context**
Reported in URL


Question: Segfault when importing sklearn as well, osx-arm64?

They seem to bundle `libomp.dylib`.

Following should fix this issue

diff --git a/recipe/build_pytorch.sh b/recipe/build_pytorch.sh
index cd27be0..c6cb567 100644
--- a/recipe/build_pytorch.sh
+++ b/recipe/build_pytorch.sh
@@ -1,13 +1,9 @@
 set -x
-if [[ "$megabuild" == true ]]; then
- source $RECIPE_DIR/build.sh
- pushd $SP_DIR/torch
- for f in bin/* lib/* share/* include/*; do
- if [[ -e "$PREFIX/$f" ]]; then
- rm -rf $f
- ln -sf $PREFIX/$f $PWD/$f
- fi
- done
-else
- $PREFIX/bin/python -m pip install torch-*.whl
-fi
+source $RECIPE_DIR/build.sh
+pushd $SP_DIR/torch
+for f in bin/* lib/* share/* include/*; do
+ if [[ -e "$PREFIX/$f" ]]; then
+ rm -rf $f
+ ln -sf $PREFIX/$f $PWD/$f
+ fi
+done

Segfault when importing sklearn as well, osx-arm64
### Solution to issue cannot be found in the documentation.

- [X] I checked the documentation.

### Issue

This issue affects the conda-forge version too.

The solution is `export PACKAGE_TYPE="conda"` in the recipe. This was the change that introduced the bug. It puts an openmp into the site-packages directory and uses that one, if the above env variable isn't set.

Some more info:
otool -L on the libraries to see what they’re linking against:

pytorch v2.2:

/Users/dpetry/miniconda3/envs/test_torch_torchonly/lib/python3.12/site-packages/torch/lib/libtorch_cpu.dylib:
 ...
 @rpath/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
 ...

pytorch v.2.3:

/Users/dpetry/miniconda3/envs/test_torch_torchonly/lib/python3.12/site-packages/torch/lib/libomp.dylib:
 @rpath/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
 ...

/Users/dpetry/miniconda3/envs/test_torch_torchonly/lib/python3.12/site-packages/torch/lib/libtorch_cpu.dylib:
 ...
 @loader_path/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
 ...

### Installed packages

shell
n/a

### Environment info

shell
n/a

The referenced issue ( URL ) points to a user recreating the bug with "anaconda's default channel". We do not use "anaconda default channel" packages.

If you can recreate with conda-forge only channel:

please provide all the requested outputs from the issue template. they are critical when troubleshooting this.

conda create --name pt python=3.11 pytorch scikit-learn --channel conda-forge --override-cannels

then we can start to investigate.

I also worked on a similar issue 2 months ago
 URL

so getting the full output from you would really help troubleshoot things.

yes, can recreate with c-f channel. will provide more shortly

Thank you for the detailed installation and recreation.

So from your experiments I can see that:

* Pytorch 2.4.0 no issue
* Pytorch 2.3.0 issue

Can you try to see if

python -c "import numpy; import torch; torch.zeros((1024, 1024), dtype=torch.uint8)"

recreates things with pytorch 2.3.0 build 0.

I think I hit this issue with: URL

We fixed it in 2.3.1 build 1 URL

I'm honestly not sure how far back to mark the pytorch M1 builds as broken.

If you do the investigation, all with the conda-forge channel channel only, and show your work, I can try to merge in a PR that you make
 URL

I unfortunately do not have an OSX arm machine, so it isn't easy for me to test.

ok, cool. I can do this if you feel it's useful to your users? I maintain the pytorch recipe at anaconda, so this wouldn't be work that benefits me directly - I just thought making you aware of it might be helpful. But I'm happy to do it if you feel it's valuable. If it's low-priority and it's just making more work for you then no worries. Let me know :)

And in general, let me know to what extent collaboration would be welcome from you.

its not low priority as much as "I can't really test it".

it is, but it might not be worth your time.

Generally speaking, i miss the days where the recipes were similar between conda-forge and anaconda. sharing ideas was much simpler. However, with things diverging, i'm not sure the extent that it is possible. Where is your pytorch recipe stored today?

My general understanding was that collaboration was still happening on the `python-channel` which introduced some small compatibility shims:
 URL

I'm open to adding such shims. We have been having trouble with the aarch64 builds and that has been weighing me down.

Help there would be greatly appreciated! See URL

Our recipe is here. It diverges a fair amount but there's some stuff in common too (I want to pull across the single building of libtorch stuff).
It'd be great to collaborate on v2.5.0 of pytorch, and I can have a look at the aarch issue if I get time before then.

@hmaarrfk what was the aarch64 issue you were having specifically? I couldn't really tell from a glance over the issue. Are you still having them?

""PyTorch was compiled without NumPy support" error when running on Linux aarch64 + CUDA (on NVIDIA GH200) using the conda-forge build of PyTorch 2.4.0."
was it this?

Thanks. We heard about pytorch dropping their conda package builds. How's the support for win and triton at the moment? as I understand you don't have either, we can help with that if you need.

You can find a list of issues URL

I've tagged them all. I mean its all volunteer led. Comment on the appriopriate one, make a PR if you want to see it improve.

So help where you can. To companies that are interested in windows support, I simply suggest volunteering to build things. It goes a long way.

Oh, the conda-forge CI can't handle the pytorch build for windows? hmm ok.

Can one mark only the `macosx-arm64` builds for 2.3.0 as broken? If so, that seems like the one action item left here. If not, probably best to close this as fixed.

I'm happy to click the merge green button if one makes the PR listing the exact package names.

the bug is that users with osx-arm can accidentally install 2.3.0 and crash their system.

I've filed URL ; hope I got it right.

Used the following to get the package list:

conda search 'libtorch=2.3.0[subdir=osx-arm64]' | awk '{ print "osx-arm64/" $1 "-" $2 "-" $3 ".conda" }'
conda search 'pytorch=2.3.0[subdir=osx-arm64]' | awk '{ print "osx-arm64/" $1 "-" $2 "-" $3 ".conda" }'
conda search 'pytorch-cpu=2.3.0[subdir=osx-arm64]' | awk '{ print "osx-arm64/" $1 "-" $2 "-" $3 ".conda" }'

Segfault on M1 Macbook
### Solution to issue cannot be found in the documentation.

- [X] I checked the documentation.

### Issue

mamba create --name pytorch python=3.10 pytorch numpy --channel conda-forge --override-channels
mamba activate pytorch
python -c "import numpy; import torch; torch.zeros((1024, 1024), dtype=torch.uint8)"

### Installed packages

shell
% mamba list
# packages in environment at /Users/mark/miniforge3/envs/pytorch:
#
# Name Version Build Channel
bzip2 1.0.8 h93a5062_5 conda-forge
ca-certificates 2024.6.2 hf0a4a13_0 conda-forge
filelock 3.15.4 pyhd8ed1ab_0 conda-forge
fsspec 2024.6.0 pyhff2d567_0 conda-forge
gmp 6.3.0 h7bae524_2 conda-forge
gmpy2 2.1.5 py310h3bc658a_1 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
libabseil 20240116.2 cxx17_hebf3989_0 conda-forge
libblas 3.9.0 22_osxarm64_openblas conda-forge
libcblas 3.9.0 22_osxarm64_openblas conda-forge
libcxx 17.0.6 he7857fb_1 conda-forge
libffi 3.4.2 h3422bc3_5 conda-forge
libgfortran 5.0.0 13_2_0_hd922786_3 conda-forge
libgfortran5 13.2.0 hf226fd6_3 conda-forge
liblapack 3.9.0 22_osxarm64_openblas conda-forge
libopenblas 0.3.27 openmp_h6c19121_0 conda-forge
libprotobuf 4.25.3 hbfab5d5_0 conda-forge
libsqlite 3.46.0 hfb93653_0 conda-forge
libtorch 2.3.1 cpu_generic_hf1facdc_0 conda-forge
libuv 1.48.0 h93a5062_0 conda-forge
libzlib 1.3.1 hfb2fe0b_1 conda-forge
llvm-openmp 18.1.8 hde57baf_0 conda-forge
markupsafe 2.1.5 py310hd125d64_0 conda-forge
mpc 1.3.1 h91ba8db_0 conda-forge
mpfr 4.2.1 h41d338b_1 conda-forge
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
ncurses 6.5 hb89a1cb_0 conda-forge
networkx 3.3 pyhd8ed1ab_1 conda-forge
nomkl 1.0 h5ca1d4c_0 conda-forge
numpy 2.0.0 py310h52bbd9b_0 conda-forge
openssl 3.3.1 hfb2fe0b_0 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
python 3.10.14 h2469fbe_0_cpython conda-forge
python_abi 3.10 4_cp310 conda-forge
pytorch 2.3.1 cpu_generic_py310hb190f2a_0 conda-forge
readline 8.2 h92ec313_1 conda-forge
setuptools 70.1.0 pyhd8ed1ab_0 conda-forge
sleef 3.5.1 h156473d_2 conda-forge
sympy 1.12.1 pypyh2585a3b_103 conda-forge
tk 8.6.13 h5083fa2_1 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xz 5.2.6 h57fd34a_0 conda-forge

### Environment info

shell
% conda info

 active environment : pytorch
 active env location : /Users/mark/miniforge3/envs/pytorch
 shell level : 2
 user config file : /Users/mark/.condarc
 populated config files : /Users/mark/miniforge3/.condarc
 /Users/mark/.condarc
 conda version : 24.5.0
 conda-build version : not installed
 python version : 3.10.14.final.0
 solver : libmamba (default)
 virtual packages : __archspec=1=m1
 __conda=24.5.0=0
 __osx=14.5=0
 __unix=0=0
 base environment : /Users/mark/miniforge3 (writable)
 conda av data dir : /Users/mark/miniforge3/etc/conda
 conda av metadata url : None
 channel URLs : URL
 URL
 package cache : /Users/mark/miniforge3/pkgs
 /Users/mark/.conda/pkgs
 envs directories : /Users/mark/miniforge3/envs
 /Users/mark/.conda/envs
 platform : osx-arm64
 user-agent : conda/24.5.0 requests/2.32.3 CPython/3.10.14 Darwin/23.5.0 OSX/14.5 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
 UID:GID : 503:20
 netrc file : None
 offline mode : False

Recreatable with:

* Pytorch 2.3.1 + numpy 1.26.4
* Pytorch 2.3.0 + numpy 1.26.4
* Does not happen with 2.1.2 + numpy 1.26.4

And now with lldb

% lldb -- python -c "import numpy; import torch; torch.zeros((1024, 1024), dtype=torch.uint8)"
(lldb) target create "python"
Current executable set to '/Users/mark/miniforge3/envs/pt/bin/python' (arm64).
(lldb) settings set -- target.run-args "-c" "import numpy; import torch; torch.zeros((1024, 1024), dtype=torch.uint8)"
(lldb) run
Process 58336 launched: '/Users/mark/miniforge3/envs/pt/bin/python' (arm64)
Process 58336 stopped
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x540)
 frame #0: 0x000000010086ef94 libomp.dylib`__kmp_suspend_initialize_thread + 32
libomp.dylib`:
-> 0x10086ef94 : ldr w8, [x0, #0x540]
 0x10086ef98 : nop
 0x10086ef9c : ldr w9, 0x1008a1308 ; _MergedGlobals + 8
 0x10086efa0 : add w20, w9, #0x1
 thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x540)
 frame #0: 0x000000010086ef94 libomp.dylib`__kmp_suspend_initialize_thread + 32
libomp.dylib`:
-> 0x10086ef94 : ldr w8, [x0, #0x540]
 0x10086ef98 : nop
 0x10086ef9c : ldr w9, 0x1008a1308 ; _MergedGlobals + 8
 0x10086efa0 : add w20, w9, #0x1
 thread #4, stop reason = EXC_BAD_ACCESS (code=1, address=0x540)
 frame #0: 0x000000010086ef94 libomp.dylib`__kmp_suspend_initialize_thread + 32
libomp.dylib`:
-> 0x10086ef94 : ldr w8, [x0, #0x540]
 0x10086ef98 : nop
 0x10086ef9c : ldr w9, 0x1008a1308 ; _MergedGlobals + 8
 0x10086efa0 : add w20, w9, #0x1
 thread #5, stop reason = EXC_BAD_ACCESS (code=1, address=0x540)
 frame #0: 0x000000010086ef94 libomp.dylib`__kmp_suspend_initialize_thread + 32
libomp.dylib`:
-> 0x10086ef94 : ldr w8, [x0, #0x540]
 0x10086ef98 : nop
 0x10086ef9c : ldr w9, 0x1008a1308 ; _MergedGlobals + 8
 0x10086efa0 : add w20, w9, #0x1
 thread #8, stop reason = EXC_BAD_ACCESS (code=1, address=0x540)
 frame #0: 0x000000010086ef94 libomp.dylib`__kmp_suspend_initialize_thread + 32
libomp.dylib`:
-> 0x10086ef94 : ldr w8, [x0, #0x540]
 0x10086ef98 : nop
 0x10086ef9c : ldr w9, 0x1008a1308 ; _MergedGlobals + 8
 0x10086efa0 : add w20, w9, #0x1
Target 0: (python) stopped.
(lldb)

OMP_NUM_THREADS=1 python -c "import numpy; import torch; torch.zeros((1024, 1024), dtype=torch.uint8)"

seems to be ok with things.

I'm somewhat afraid of setting package type conda due to the fact that their commit says that it is needed for torch compile......

But I would rather avoid this disastrous failure mode for others.

I expect that many simply haven't been able to update to 2.3.0 due to some ecosystem in compatibility because of multiple ongoing migrations, but I was able to by carefully picking and choosing packages and some on my team had reported the failure last week


Question: N64 Smash Bros Character select bug?

For those that will not fix their N64 Digital install and that liked the imperfect workaround here a build of v1.8 that reintegrate it.

v1.8-push_pull_port1_hw1_n64.zip

N64 Smash Bros Character select bug
### BlueRetro firmware version

25.01

### BlueRetro firmware specification

HW1

### BlueRetro firmware variant

System specific

### BlueRetro hardware type

External adapter dongle (1 port only)

### Manufacturer

RetroTime

### System used

Nintendo 64

### Bluetooth controller brand & name

NSO N64 Controller, PS4 Controller

### What is the problem? (only list ONE problem per report)

During character selection in Smash Bros. or Smash Remix, whether using an everdrive or original cartridge, the player character selection will spontaneously reset, clearing your currently selected character and instantly returning the "P1" coin to your floating hand, and returning the floating hand to the bottom left of the screen. It happens often enough that character selection is very difficult and can only be accomplished after multiple attempts and moving on to stage selection faster than the issue can occur. This is the only scenario I see this kind of behavior, and the issue is nonexistent with a standard controller.

### What did you expect to happen?

to be able to select a character normally

### Attach BlueRetro debug trace here

_No response_

Did that adapter ever work? or it's something new after you changed something?

I’ve had the adapter since ‘23 and I’ve always had this issue with every firmware I’ve used

Ok I just tried Smash on my Everdrive and I don't have this issue.

Do you some kind of HDMI mod with a OSD? Those are known to create inteference with BlueRetro.

Ahh yes I do. I have the N64 digital hdmi/RGB mod installed. Is that something unlikely to be fixed down the line through a firmware update? (Either blueretro or retro GEM)

See this post
 URL

You can do as a document it or try get a replacement flex from the hdmi mod vendor.
Or you can simply disconnect the OSD line going to the pif

Gitches on Nintendo 64 with N64 Digital while using Controller/Rumble Pak / F-Zero
I’ve enabled the controller pak manager through the web interface and formatted it but a number of games prompt that there is a problem with the pak on launch. One example is Tony Hawk Pro Skater 1. As a separate issue, F-Zero does not detect the blueretro device as a controller.

Console is modded with N64 digital and using latest version of the blueretro firmware. Unrelated, but with two NSO 64 controllers the number returned from the controller test tool for the right axis seems to be a bit higher than the benchmark. What setting in the advanced config should I adjust? Two different NSO 64 controllers return different values on the test tool, so there must be some variance at the factory with those sticks.

Try to format via a laptop if you used a phone, or just let tony hawk format it when prompted.

If detection fail you can simply press reset on console, also games give up too fast.

Play with the LX, RX maximum value in web config to fine tune joystick.

IMHO this is a waste of time, that test tool make people paranoid. Most games are programmed with a good play room.

Thanks for replying! I've tried the suggestions but F-Zero does not work at all with a blueretro device and Tony Hawk will not recognize the controller pak. Testing a random sample of other controller pak games shows a major issue with the current integration. I've also experienced odd rumble behavior. On Goldeney 007, the rumble will not work correctly when firing. It gives a brief rumble and then stops when firing, when it should continue to rumble when firing. This is a great product, and hopefully you'll be able to circle back around to the N64 to finish polishing the feature set. Thank you!

You might have some build related issues because I personally got none of those issues.

Sound like your data line is glitching somehow.

This was confirmed from others using the adapter purchased from Tindie so it is likely the build. I’ll follow up with the seller.

If you can ask others to report feedback in this GitHub issues.

Everyone with issue should list the following information if possible:
* N64 motherboard revision
* List of internal mods installed
* Other external peripheral used

It could be problem with the dongle itself but it could also be some interoperability issue with some motherboard or some other mods.

I had mempak issues myself before, but they were resolved when I used a PC instead of a phone to change the config. These issues occured both on my adaptor as well as on an ESP32 Devkit which I used to compare functionality.
Regarding the other issues, both with a ESP32 Devkit as well as my own Adaptors I had sporadic rumble issues, close to the ones that you described.
Nothing gamebreaking though, the core functionality always worked. To be honest I always accepted it as minor issues in a great open source project, maybe depending on different controllers, console revisions and mods used. In the issues section on GitHub there are also several topics related to these small issues, so I did not look more into it and accepted it as small software bugs that don't really affect me.

Now of course I don't want to blame you @darthcloud for anything in your amazing project and I want to offer the most for everyone who bought my Adaptor, so I will look further into it.

Because I just moved last week, it will take more time than usual, I barely even have a TV set up at the moment, and my N64 consoles as well as the Adaptors are still in boxes.

I will try to get to it this weekend.

@bixxewoscht Anything is possible off course but the N64 is the original system BlueRetro was developed for and that I actually use myself a lot. I gone though GoldenEye a few time ever the last 3 years and never noticed any rumble issues. The way it is implemented makes the interface very robust as it is using the ESP RMT peripheral, (so no software bit bang). What you guys describe should not happen at all.

I'm using my own BlueRetro Devkit when playing but I did test with a few clone ESP-DevKit-C in December and this spring and didn't got any controller pak error myself. Controller pak was tested by going through Turok 1 and Tony Hawk 1.

The issue sound like the data line is glitching.

My own board got ESD protection on each line using a TPD6F002DSVR chip. This got an equivalent resistance of 100ohm. No idea if this help or not.

Maybe chip me one of your dongle for me to test.

I play with early N64 rev mostly but got a few more now that I could try on too. I got no mod in my console.

@Taurus1100 / @Taurus1099 I got your adapter today and made some quick test and it look to be working fine for me.

See video:
 URL

I tested with your F-Zero X and TH1 cartridge with a NUS-CPU-04 and NUS-CPU-08-01 motherboards. Both work fine.

I got others N64 mobo in storage will dig them this weekend and report back.

Meanwhile can you confirm which mobo your N64 got? Did you install any mods? Are you using an original AC adapter?

Thanks for taking a look. My console revisions is NUS-CPU-08-01 (modded with N64 Digital) and I'm using the original power supply. I also tested with an additional power supply and no change. When you are able to do further testing, are you able to get into a game on F-Zero? I've seen the game respond to controller inputs for 10-15 seconds but eventually it stops completely. The N64 Digital menu is still accessible however. Also, any issues with the rumble in Goldeneye?

I think its likely the N64 digital that interferes with the pad data lines. I can test more this weekend but its likely all fine.

I don't know much about N64 digital but I think you could remove the one connection for pad and it would likely work fine. But you would lose the OSD access.

Must be related to the N64 Digital. I noticed a similar issue on another open source project:

 URL

I tested with my N64 with N64 Digital and had the same issue with F-Zero. Could not test with my stock N64 yet. But in my other tests I remember there were no issues with the game on a stock N64.

Thank you both for investigating further.

Holen Sie sich Outlook für Android
Sent: Thursday, May 19, 2022 12:52:29 AM
To: darthcloud/BlueRetro ***@***.***>
Cc: bixxewoscht ***@***.***>; Mention ***@***.***>
Subject: Re: [darthcloud/BlueRetro] bixxewoscht's dongle gitches Nintendo 64 Controller Pak / F-Zero (Issue #264)

Must be related to the N64 Digital. I noticed a similar issue on another open source project:

Ryzee119/usb64#42

n64_test_push_pull.zip

This is untested but it change the data line from open-drain, to push-pull. Maybe that will help if the issue is on the TX side.

Everything is now working perfectly! The added ability to get vibration feedback on switching between rumble/memory pak is very helpful. Is there any functionality lost by changing the data line?

Hi @darthcloud, I'm the guy that slid into your DM's. Similar to the above user, I can confirm that the new firmware fixes all of the issues I was having. I played with it for quite a while today and exercised all of the main functionality and everything worked great. Thaks for finding a workaround so quickly... I was excited, then super bummed, and now excited again... this is a really neat implementation and will be my goto N64 setup going forward!
________________________________________________________________________________________________________________
Just in case it is useful info for future reference, when I _was_ having problems it was the following setup:
NUS-CPU-08-01 with N64Digital mod
stock PSU
Switch Pro controller
v1.5.1 of your firmware
@bixxewoscht receiver

What was happening (this was only for P1 on my modded system; P2 worked fine and my stock NUS-CPU-08-01 worked fine)
Controller pak emualation wasn't working at all (games either wouldn't recognize it or would report it corrupted)
Every few seconds an uncommanded controller input [Dpad down + Dpad right + main joystick (1,-1)] would register

I can confirm this firmware seems to resolve the issues I was having on my n64digital console as well, both with F-Zero X and overall memory pak issues.

I got an N64 digital this week and was finally able to investigate first hand.

I wasn't really a fan of the workaround I made to fix it, the N64 SI is means to have device with open drain IO. So switching to push-pull is not really ideal IMHO.

This is really an issue caused by the N64 Digital mode and I think the fix should be an HW on it.

To fix it I added a 74AHCT125N buffer between the the PIF pin 16 and the N64D ribbon COM pad

This did the trick F-Zero X menu doesn't glitch with BlueRetro and the N64D OSD still work as before.
kicad_qEtttH5Odq

The best would be to design a ribbon with a pad for the tiny 1 gate variant 74AHCT1G125W5-7

PXL_20220813_205732992
PXL_20220813_205811684

Will you please leave the software fix in place for those of us without skills to mod our systems? I would have to pay for this work to be done :-(

Problem is this fix doesn't work for everybody, and it actually break it for some people without a N64D.

Old FW stay available so you can keep what you currently have, and if any major update occur I can always post a special version in this thread

Quick update PixelFX designed a small flex to fix this issue. I tested it and it work great.
PXL_20220927_000325996

I'll update this issue once info on how to get it is available.

@darthcloud Is this fix the same for both HW1 and HW2? ie for both external and internal (with detection) installs?

@darthcloud If I'm reading correctly, you prefer Dan's flex cable fix to your 74AHCT125N buffer method for electrical reasons?

They are both using 125 buffer, it's same thing except flex look nicer

Hello there, is there also a fix planned for UltraHDMI users? I updated to the latest firmware and now the system does not detect the controller at all anymore

Yikes! I have just ordered one of the dev kit chip to do an internal install on my UltraHDMI modded N64. Has anyone reported a similar issue bundled with that mod? If so, a new flex cable will have to be developed, and as nice as Marshal is, he's only one guy and has a busy life. The chances of him developing one is slim, unless someone else steps in and develops one of their own. But again, has anyone with a UltraHDMI mod had this problem? Should this deveate into a different thread?

This flex doesn't seem available anymore?

I am having an issue with my N64 digital modded system along with the blue retro external mod. My NSO and 64 controller will sync. And then randomly will disconnect mid game. Any work around?

Dan of pixelfx/blackdogtech said there will be more flex cables in the future, but without a date. I would guess in the next 5 months. Or you can build your own circuit w/buffer with the info above. If you go the FFC route, you will need to solder surface mount components onto the FFC. Below I've listed the component to DIY, and the SMD components Dan originally specified for his flex cables, but he removed the FFC web pages so it's possible they change in the future. There's no difference between the two, the FFC simply looks nicer if you're into staring at the innards of your console. The DIY option will be easier to solder.

Buffer for DIY circuit - TI SN74AHCT125N: URL or URL
FFC N64 Buffer - Nexperia 74AHC1G17GVH: URL or URL
FFC N64/PS1 Ceramic Capacitor(s) - 0.1uF 10V 0402: URL or URL
FFC PS1 Buffer - TI SN74HCS125QPWRQ1: URL or URL

Please double check above and the FFC listing whenever it returns to ensure I made no mistakes.

So I should just need to order the buffer for diy (since i dont care what it looks like) and them just solder like example above? Or do I also need capacitor(s)

Yeah, DIY is easiest and fastest. I don't see any caps in the circuit, but @darthcloud is the expert, if he sees this.

My recommendation is to avoid firmware 1.8 and above or, alternatively, stop using BlueRetro for N64. We shouldn’t have to further mod our consoles.

I had a customer send me in the flex kit and his N64 that I put the HDMI kit in. This flex seems to kill the ability for a normal N64 controller to enter the OSD?

@L10N37 It shouldn't do that, probably problem with flex install.

ah it was like 3/ 4am and pulled 2 all nighters in a row, i just missed the 5v connection

Any latency or disadvantage to this work around as opposed to adding a buffer?

The fix is integrated in firmwares prior to 1.8 but it was removed in the main branch.

Its a hardware bug and the workaround only work for half people with n64 digital, and break half people without one.

The real fix is the pixelfx patch cable.

n64 digital hardware bug, this is why they provide a flex patch

As an ultra hdmi user, I suffer the same problem.

And I know for sure that there won’t be a fixed flex cable, at least not from the mod creator.

so basically I have no choice but to either switch to N64Digital or use those workarounds. I hope those will be provided in the future versions as well. Thank you

You could also look at the Flex/DIY fix documentation and adapt it for Ultra HDMI. There shouldn't be too much of a difference. But I don't know it exactly since I've never looked at Ultra HDMI.

You just need to do as I explained in my previous post in this thread
 URL

@darthcloud Ok, let me see if I get this one strait. The only wire you are intermediating is the N64D ribbon cable to the PIF 16. You aren't messing with any of the other connections? I'm asking because I'm finding the similarities between the N64D and the UltraHDMI. The UHDMI has a ribbon cable that I believe you attach to the same spots as the CON connection on the N64D.

ezgif com-gif-maker
(not my photo, taken to show example)
So would that mean that you would connect the UHDMI ribbon -> Pin 2 Pin 3 -> (whatever pin depending on your motherboard)? Cause that might be the solution for the select few that own UltraHDMI sets.

@Sowden Yeah just piggy-back a 74AHCT125N on top of the PIF chip like I did (connect 3V3 & GND to pif pin directly) then the pin label N64D_COM (pin3 ) is the one you plug your ultra HDMI cable to. and pin 2 goes to the pif controller 1 pin. Don't forget to GND all the needed pins as done in my schematic.

Oh, GND Pin 1 -> PIF 14 and 3.3 v Pin 14 -> PIF 28? So the 74AHCT125N has four connections in total, is that correct?

Just wanted to pop in here and see if my issue might be related. I've got an N64 with the Gamebox Systems 64HD. When I try to use my BlueRetro adapter (from Retrotime) in controller slot 1, I am getting graphical glitches only on my HDMI out (regular composite output is unaffected). Other controller slots do not have the issue. It's almost as if (and this is just a guess) the OSD menu is being interacted with even though I haven't brought it up. I haven't tried your firmware posted here yet though, I'll give it a try tomorrow.

From what I heard from other users (I don't have a 64HD) the Gamebox 64HD definitely has the same issue as N64Digital and other mods that tap into the controller lines. Unfortunately the known fix was not implemented from the get go on the N64HD.

I updated my device with v1.8-push_pull_port1_hw1_n64.zip and it did not resolve the graphical glitches :(

To be clear, with the 64HD I am NOT getting issues with F-Zero or Tony Hawk as reported here for N64 Digital and Ultra HDMI mods.

I am just getting graphical glitches when my BlueRetro adapter is in controller port one. The glitches usually look like a bunch of random colors flash across the bottom half of the screen. The game I have found it most noticeable on is Snowboard Kids. Here is a video: URL

I heard from people with a N64 HD that it has problems with signal integrity as it is, without using blueretro simultaneously. Reason is most likely the flex-pcb design thats not up to industry standards (ground return path). So likely when using blueretro this amplifies the issue.

Yea, I have noticed very infrequently some visual issues, but mostly it is fine. BlueRetro is increasing the frequency by like 100x or 1000x though.

I've ordered the flex cable, I am having issues where the OSD is occasionally inaccessible. Where can I find the installation instructions? I have a friend who is experienced at soldering, hope he can help.

I get graphical glitches, video desync, and signal loss from my N64 Digital via HDMI using Blue Retro. Is this problem related?

Anyone know if this issue was resolved with the Retro Gem? I would assume it was but haven't read anything to confirm.

Doubt it. N64 RetroGem = N64 digital. Same device just different branding.

From what I saw in this thread it was confirmed issue was a n64 digital hardware bug. Pixelfx were selling a flex cable specifically for n64digtal, but i see no mention of it for the retrogem.
As for N64 RetroGem = N64 digital. They use different firmware. Also the hardware design and components are not exactly the same. Pixelfx were aware of this bug. Hence why I'm curious if it was taken into consideration. I'll ask on their discord.

@darthcloud I know this is a workaround and not a real fix, but any chance of getting a firmware 1.9.1 with the same modification to change the data line from open-drain to push-pull? I was hoping to use my 8BitDo Mod Kit controller with my BlueRetro HW1 adapter, and there were some related fixes in 1.9 that I was hoping to take advantage of.

I have an N64Digital HW1, and besides not being able to find the flex cable that fixes the core problem, my soldering skills are horrendous and I'd rather not pay more to have someone do the installation (the original N64Digital mod and install was already pretty expensive).

I understand if this use-case isn't something you can keep supporting. Thanks!

This is my quick'n'dirty implementation:
PIF N64Digital fix
IMG_5688

For anyone else with an N64Digital, I found the BlueRetro flex still available on PixelFX's site. It's just under the legacy replacement parts page here: URL

I haven't bought it myself and honestly, I'm unsure if I will since I know I'll have to send it again for an install. I got mine installed back in Summer 2022 by Rainy Day Retro (wonderful mod shop, highly recommend), but at this point... I think I may just resort to playing Controller Pak games on my Mister Pi.
And I don't trust myself with installing it after a botched GBA SP USB-c mod install in which I ripped some pads in pursuit of better alignment. and the high temps needed for lead-free solder.