Question: Always-onscreen keyboard bar?

Hi @sudomain , I am working on full re-implementation of virtual keys. The new layout follows the same style as Termux. (There are some differences as the use case is different for AVNC).
- 2-row layout similar to Termux
- Meta keys are automatically released on next key press. Long-press to lock meta keys, as described in this particular issue.
- There are three swipable "pages":
 - First two pages contain keys
 - Similar to Termux, third page contains an editbox. It allows you to have full-fledged Android editing experience in a session. When finished, you can send the whole text to server in one go.

First page:
Screenshot_2024-04-04-21-51-23-702_com gaurav avnc debug-edit

Edit box:
Screenshot_2024-04-04-21-52-25-104_com gaurav avnc debug-edit

---

While there are still some rough edges, I would appreciate if you guys can test it.
Test APK: URL

cc @dp-alvarez

Always-onscreen keyboard bar
Firstly, thank you a million for this app. I've been looking for an open source alternative to the Remotix app and no one has ever even come close until yours, and I've looked regularly for a long time. Amazing work, I've officially uninstalled it and haven't looked back

But anyway one thing that'd be really great and cool and epic would be an option to keep that bottom keyboard row on screen at all times. I'm always reaching for it and it would be nice to just keep it there

Glad you find AVNC useful! I have tried to make it as seamless as possible.

There is already an option to automatically open "virtual keys" with the soft-keyboard (`Settings => Input => Virtual keys`). But if you are talking about showing them when connection is started, that can be implemented too.

I do have that option checked, i just would like it on screen by default. Bonus points for an option to have a "show keyboard" button on there too ;)

I will change the `Open with keyboard` option to a list with three options: Open manually, Open automatically, and Open with keyboard.

Hi,

Thank you very much for developing this awesome software.

While we're on the subject of on-screen keyboard and when and how to open it, I just wanted to share my experience here:
- when I'm using AVNC on my phone, for quick sessions, then, the two-finger-tap to open keyboard works well, although I do need to tap and tap and tap to get it right (maybe this action could be backed up by a hardware key, volume-up or volume-down or something)
- when I'm using AVNC on my tablet, which has a mouse attached to it, for longer sessions, then, I always need android keyboard + virtual keyboard open, so I find myself two-finger-tap-tapping all the time and then I realized that it would be super useful if we could have a global option "auto open keyboard on session start"

So I found this thread and figured, best to join in here.

Thanks.

AVNC rocks!

Although a slightly different request, it's still similar enough that I'm going to share my feedback here.

When using a tablet with a thick and strong case, it's really hard to swipe from the edge of the device, as the case goes around the screen. Thus, it takes too many tries to open the sidebar.

I would suggest to incorporate the sidebar buttons into the keyboard bar. Being able to quickly toggle the touchscreen/touchpad behavior without accidentally moving the mouse cursor would be very welcome.

Likewise, it would be nice to have virtual buttons for scrolling up/down and middle-click.

I would like to have the keyboard bar always visible.

I understand this isn't the original request of this issue, but it's related and I didn't want to open another issue for something so similar. (Also, I typed this on AVNC with touch keyboard and a laggy VNC connection.)

Hi all, a bit late, but next version will contain many improvements related to virtual keys:
- Two-row layout, similar to Termux
- Dedicated button to toggle Android soft-keyboard
- Persistence of visibility across sessions: Once opened, virtual keys will continue to automatically open on each connection (same goes for closing them)
- Other changes like a textbox to allow full-fledged editing experience, and long-press to 'lock' meta keys

Preview:
image

Test APK: URL

oh this is lovely, just tried it! Thank you so much - I've actually since switched back to Remotix because of these issues and I am happy to come back with these changes :)

Great! I had hoped to implement these sooner but couldn't get enough free time to properly fix all of these (re #117 , #187) .

Two-row layout is great, still one feature request:
on my tablet, this virtual keyboard is only half of my screen width, which is wasting of space.
How about make it using the complete with, to also have the "extended keys" easy reachable at the same time?

Yeah, its more optimized for mobiles. Making these more configurable is already on my todo list.

Feature request: change behavior of virtual (modifier) keys: short press keys to make them apply to only next key event, long press to have the key be "sticky".
Hello,

Could the modifier keys (Shift, Ctrl, Alt, and Meta (Windows)) be made so that a long press causes them to become "sticky" (I.e. sticky keys) or applied to every key press until they're turned off with another press please? This is the current behavior of the short press of a modifier key. The reason I ask is because I'd also like the short press of modifier keys to apply only to the next key press, and then deactivate. This is similar behavior to Termux's extra-keys row, but also on many Android soft keyboards short pressing Shift tends to only capitalize the next letter, whereas long pressing Shift tends to activate caps lock.

That was already on my mind, but then the current implementation got slightly more complex. So I decided to get this working for now, and put custom configuration on hold. Maybe someday it will be implemented, but I don't have any immediate plans for it.

That said, is there any key missing from current layout?

v2.5 is now available with this change.
And I will let you know when the keys are more configurable.

Hi @sudomain I am working on allowing users to configure key layout. Instead of a file, I have added a small UI to customize keys.

Test APK: URL
Go to `Settings => Input => Virtual Keys => Customize keys` to make changes.


Question: Should there be optional `undesirable_object_name()` linter?

I wrote something similar to this, the issue is there are several thousand names to check which would make the linter relatively heavy

Should there be optional `undesirable_object_name()` linter?
A number of potential object names can cause trouble because they also represent commonly used function names.
Here are a few that come to mind (can definitely be expanded based on feedback):

r
t, c, df, I # (from `{base}` package)
data # (from `{utils}` package)

In edge cases, these produce the cryptic `object of type 'closure' is not subsettable` error:

 r
df Error in df$x: object of type 'closure' is not subsettable

 Created on 2023-01-09 with reprex v2.0.2

So should there be an `undesirable_object_name()` linter that would caution users to avoid such variable names?
Examples that would produce lints:

r
df Error in c(2, 2): unused argument (2)

 Created on 2023-01-09 with reprex v2.0.2

Definitely related, but this issue has a much narrower scope in mind.

Checking against every function name in default packages (stats, graphics, grDevices, utils, datasets, methods, base) is going to be an expensive check, so I thought restricting such a check to a few functions might be a good compromise.

Maybe we keep this open until we decide either way in #376?

I have an existing implementation that used something like the top 100-most overloaded symbols (at least in our internal code)

find variables which reuse names from functions or variables in base or recommended packages
Although the code in R itself doesn't follow its own recommendations, it would be nice to lint for times when a variable is defined which overrides the name of something in a base or recommended package. I imagine it would be easy to string match against the contents of those namespaces.

E.g., I would love a lints for:

version <- "version is a 'simple.list' in the base package"
names <- c("alice", "bob") # oh no, names is a function in base

This can lead to problems, e.g. when editing the variable declaration, but not updating subsequent uses, since the symbol will still be found, but will often be for a closure, giving opaque errors.

This would apply to both variables defined within functions, and function arguments, so the following would also be good to lint:

my_fun <- function(data.frame = cars) head(data.frame)

Maybe reusing logic from the `object_usage_linter` and checking for collisions of all of the found symbols within the base & recommended packages would be a lightweight way to implement this.

That way we wouldn't need a list of all bad object names, but instead dynamically check the (presumably fewer) used symbols against a bunch of loaded namespaces.

See #2307. It's not as slow as you might expect (though noticeably slower than the fastest+simplest linters)

Marking this as closed by #2307. please file any residual FR as a follow-up


Question: New page that lists all Price tag proofs without any prices yet (= open for contribution)?

But I don't understand how with your current `price_tags` model proposal you can store multiple predictions ? It's missing a data JSONField, thus we could simply plug in the ProofPrediction model (or a dedicated `PricePrediction`)

New page that lists all Price tag proofs without any prices yet (= open for contribution)
### Story

Following changes in the backend - URL - users are now allowed to add prices on PRICE_TAG proofs they do now own. Thus unlocking distributed contribution.

To kickstart this, we need a page that lists all available proofs open for contribution.

### How

A list of all proofs that have the following caracteristics:
- type = PRICE_TAG
- price_count = 0
- location_id__isnull = False

And clear actions for each proof :
- view full size image
- add prices
- add prices via the contribution assistant

### Questions

- how to avoid users adding prices on the same proof ?
- how to skip proofs that are unreable or without any barcodes displayed ?
 - allow users to tag proofs ?

### Ideas

- display proofs in a random order on each load, to mitigate the risk of having 2 users simultaneously adding prices on the same proof

Displaying proofs at random is enough for now! We should just keep in mind that SQL random() doesn't scale well to huge tables (as seen on Robotoff).
I wonder whether it's better to wait for URL to be implemented in the backend, as it will make it easier from a front-end perspective:
- fetch the `price_tag`s with `status=null` randomly
- annotate (using an interface similar to the 3rd step from the contribution assistant)
- repeat

Another game could be to check that no bounding box (=item in the `price_tag` table) is missing, and if there are some, the user draw additional bounding boxes, which become available as new `price_tag`s items to annotate.

closing as we'll wait for "prices to validate list" instead of this "proofs to add list"

Add a `price_tags` table to store predictions
In #526, we've talked about how to speed up price addition using ML/AI ( URL

For price tags, the workflow seems now clear:

1. user upload their proof
2. an object detection model identifies individual price tags. The object detection model returns the coordinates of each detected bounding box, along with a confidence score.
3. we extract information from the price tags: price, EAN or product category, price type, organic or not
4. we let the user (and later other users) fix and validate the prediction

Step (2) is currently in development. Once the proof is uploaded, we would ideally run all ML models as an async job.
For each detected price tag, we need to store some information:

- the bounding box of the price tag with respect to the original proof
- the detected price
- the detected category and/or EAN
- organic/non-organic
- the price per (per kg or per unit)
- origins (for raw products)

I suggest we create a new `price_tags` table. Storing intermediate data in a new table is necessary to allow performing async processing. The user won't have to wait for models to run, and we can distribute workload (=price validation) between contributors. Besides, it allows having gold truth data evaluating and training models that extract information from price tags.

### Schema

- `id`: ID of the price tag
- `proof_id`: FK to the proof
- `created`: creation datetime
- `updated`: datetime of last update
- `bounding_box`: the coordinates of the bounding box, in relative coordinated as (y_min, x_min, y_max, x_max). The origin is the top-left corner of the image.
- `status`: The status of the price tag: either a price is already linked for this price tag (status=1), it may be waiting for approval or completion (`status=null`), or it may be invalid (`status=0`, the information cannot be read or is hidden). Only `price_tag`s with `null` status will be suggested in a "Hunger Games-like" games.

Fields pre-filled by the extraction model (currently Gemini):
- `predicted_product_code`
- `predicted_category_tag`
- `predicted_price`
- `predicted_price_per`
- `predicted_price_is_discounted`
- `price_without_discount`
- `predicted_currency`
- `predicted_labels_tags`
- `predicted_origins_tags`

Fields that are derived from predictions, validated by the user. These fields are null before the price tag is validated by the user.

- `price`
- `price_is_discounted`
- `price_without_discount`
- `price_per`
- `product_code`
- `category_tag`
- `currency`
- `labels_tags`
- `origins_tags`
- `models_info` (JSONB): extra information about the version of the models that generated the prediction, about which model was used for what, etc.

We can add a nullable `price_tag_id` in the `prices` table to keep track of the individual price tag that is behind a price.

### Workflow

When a new proof is uploaded, the price tag object detector model is run on the image of the proof. We create one element in the `price_tags` table for each detected price tag (above a fixed threshold).
For each detected price tag, we run the Gemini model on it and save the results in the `predicted_*` fields.
The `status` of the `price_tag` is null by default. Users can validate the extracted data by calling an endpoint to retrieve all `price_tag`s with `null` status. Once validated, a new price is created linked with the original `price_tag` using the `price_tag_id` foreign key.

Thanks for the detailed issue !

A few remarks:
- after having created a ProofPrediction table (in #511), isn't there similarities we could reuse here ? it might be complexifying, but I see things in steps :
 - first create the PriceTag table that stores a bounding box and a status. that could already be fed to the user in the frontend
 - then create a PriceTagPrediction that stores the result of 1 or multiple models that we run on the image. that we could re-run as well in the future if we improve the model, etc
- I see this (and the already-created ProofPrediction) in a dedicated `ml` sub-app. The backend should be able to run without these AI. Even if it will become "core" to speeding up price collection, we shouldn't make it mandatory.

We don't store multiple predictions (ex: we don't store 2 price prediction by 2 different models). On Robotoff, it turns out after a couple of years that we never needed predictions from 2 models at the same time: when a new model is trained and tested, I just delete all the predictions associated with this model and relaunch the model on all images.

Ok but I would be in favor to have the flexibility to do any number of predictions, for instance ones coming from Gemini, and another coming from our own model, and have both show up in the UI to help the user, or help us test/compare while we transition out of GenAI, no ?

If you want to keep the flexibility to have any number of predictions of the same type, it's better to have a `PriceTagProofPrediction` as you suggested!
I'm down for creating this new table then, if we plan to implement model comparison in the front-end :)

Updated schema, after the discussions above:

## Schemas

### `price_tags` table

- `id`: ID of the price tag
- `proof_id`: FK to the proof
- `price_id`: FK to the price created from this price tag, can be null.
- `created`: creation datetime
- `updated`: datetime of last update
- `bounding_box`: the coordinates of the bounding box, in relative coordinated as (y_min, x_min, y_max, x_max). The origin is the top-left corner of the image. Cannot be null.
- `status`: The status of the price tag: either a price is already linked for this price tag (status=1), it may be waiting for approval or completion (`status=null`), or it may be invalid (`status=0`, the information cannot be read or is hidden). Only `price_tag`s with `null` status will be suggested in a "Hunger Games-like" games.
- `model_version`: the version of the object detector model that created this price tag. If it was created by a human, this field is null.
- `created_by`: the name of the user who created the price tag. If the price tag was created automatically after object detection, this field is null.
- `updated_by`: the name of the user who updated the price tag coordinates. If the price tag was created automatically and never updated, this field is null.

### `price_tag_predictions` table

- `id`: ID of the price tag prediction
- `type`: type of the prediction. Currently, only one value is supported: `price_tag_extraction`
- `model_name`: name of the model. Currently, there is only one model: `gemini`
- `model_version`: version of the model. Currently, there is only one version: `gemini-1.5-flash`
- `data`: JSONB containing prediction data returned by the model. The schema of the dictionary is specific to the model.
- `created`: creation datetime

## Workflow

When a new proof is uploaded, the price tag object detector model is run on the image of the proof. We create one element in the `price_tags` table for each detected price tag (above a fixed threshold).
For each detected price tag, we run the Gemini model on it and create a new `PriceTagPrediction` object in DB linked to the PriceTag.
The `status` of the `price_tag` is null by default. Users can validate the extracted data by calling an endpoint to retrieve all `price_tag`s with `null` status. Once validated, a new price is created linked with the original `price_tag` using the `price_tag.price_id` foreign key.


Question: Mybatis INSERT foreach case SQLSyntaxErrorException?

Hi @tuichenchuxin
The value was recognized as `expr` instead of `blobValue`. I found this may be related to #16493. Could you take a look at this?

<img width="1680" alt="image" src=" URL
<img width="347" alt="image" src=" URL

Mybatis INSERT foreach case SQLSyntaxErrorException
## Bug Report

### Which version of ShardingSphere did you use?
5.1.1
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy
### Expected behavior
The following INSERT statement can run normally when calling MySQL directly, but an exception is thrown after using Sharding-Proxy.

@Insert(value = {" insert into ${tableName} (shapeid, userid, groupid, create_userid, name, type, name_alias, " +
 "stringvalue, doublevalue, intvalue, blobvalue, version, datasize, createtime, modifytime, deleted) values" +
 " " +
 "(#{item.builder.serverPID}, #{item.userID}, #{item.groupID}, #{item.userID}, " +
 "#{item.builder.fieldName}, #{item.builder.type}, #{item.builder.fieldName2}, " +
 "#{item.builder.stringValue}, #{item.builder.doubleValue}, #{item.builder.intValue}, " +
 "#{item.blobValue}, #{item.builder.version}, #{item.dataSize}, " +
 "#{item.builder.createTime}, #{item.builder.modifyTime}, #{item.builder.deleted})" +
 " "})
@Options(useGeneratedKeys=true, keyProperty="propertyExList.builder.serverID", keyColumn="id")
boolean insertBatchProperty(@Param("tableName") String tableName,
 @Param("propertyExList") List propertyExList);

### Actual behavior

### Error updating database. Cause: java.sql.SQLSyntaxErrorException: This version of ShardingSphere-Proxy doesn't yet support this SQL. 'You have an error in your SQL syntax'
### The error may exist in xxx/cn/syncserver/dao/UploadShapeDao.java (best guess)
### The error may involve xxx.cn.syncserver.dao.UploadShapeDao.insertBatchProperty-Inline
### The error occurred while setting parameters
### SQL: insert into shape_property_simple (shapeid, userid, groupid, create_userid, name, type, name_alias, stringvalue, doublevalue, intvalue, blobvalue, version, datasize, createtime, modifytime, deleted) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) , (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
### Cause: java.sql.SQLSyntaxErrorException: This version of ShardingSphere-Proxy doesn't yet support this SQL. 'You have an error in your SQL syntax'
; bad SQL grammar []; nested exception is java.sql.SQLSyntaxErrorException: This version of ShardingSphere-Proxy doesn't yet support this SQL. 'You have an error in your SQL syntax'

### Reason analyze (If you can)
I found that the following insert statement can be executed. The only difference between it and the previous statements with errors is that it does not contain ` (?,?,...)`.

[INFO ] 2022-05-23 15:20:37.162 [ShardingSphere-Command-0] ShardingSphere-SQL - Actual SQL: syncdb1 ::: insert into shape_simple (layerid, userid, create_userid, groupid, type, data, minlevel, maxlevel, version, datasize, legend, createtime, modifytime, deleted, create_version) values (0, 10232, 10232, -1, 1, x'0a1b09c240214c1bc45c4011ee44a401170b4440190000000000000000', 0, 0, 15, 120.0, '', '2022-05-23 15:18:54', '2022-05-23 15:18:54', 0, 15)

A simple PrepareStatement will also report an error. I wonder if there is a problem with the support of PrepareStatement.
In this example, the sharding column is userid.

java
 String jdbcUrl = "jdbc:mysql://localhost:3307/sharding_db?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=Asia/Shanghai";
// String jdbcUrl = "jdbc:mysql://localhost:3306/p1?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=Asia/Shanghai";
 DriverManager.registerDriver(new Driver());
 Connection connection = DriverManager.getConnection(jdbcUrl, "root", "123456");

 String sql = "INSERT INTO remark(userid, content, `time`) values (?, ?, ?)";
 java.sql.Date date = new java.sql.Date(remark.getTime().getTime());
 PreparedStatement statement = connection.prepareStatement(sql);
 statement.setLong(1, remark.getUserId());
 statement.setString(2, remark.getContent());
 statement.setDate(3, date);
 statement.execute();

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'org.apache.shardingsphere.sharding.rewrite.token.pojo.ShardingInsertValuesToken@' at line 1
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
 at com.mysql.jdbc.Util.getInstance(Util.java:408)
 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
 at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
 at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
 at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
 at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
 at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
 at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1242)

Hi @wingkingbobo, thank you for your feedback, can you provide your sharding configuration?

Thank you @wingkingbobo, can you provide your original sql, I will try to reproduce this problem.

@strongduanmu
`MySQLSyntaxErrorException` thrown by prepareStatement seems to be another problem, so I raised an issue #17888 separately.
For the current problem, I'm trying to make a simplified demo to reproduce it.

@strongduanmu
The problem is that the blob length is equal to 0.
The description of this issue is incorrect, so I opened a new one #17892.

MyBatis INSERT blob cause SQLSyntaxErrorException when blob length is 0
## Bug Report

### Which version of ShardingSphere did you use?
5.1.1

### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy

### Expected behavior
execute insert by mybatis with a blob value.

java
@Insert("INSERT INTO remark(id, userid, content, time, blobValue) " +
 "VALUES(null, #{userId}, #{content}, #{time}, #{blobValue})"
 )
 boolean insert2(RemarkPO remark);

### Actual behavior

### Error updating database. Cause: java.sql.SQLSyntaxErrorException: This version of ShardingSphere-Proxy doesn't yet support this SQL. 'You have an error in your SQL syntax'
### The error may exist in rivermap/cn/wemapsyncserver/dao/UploadShapeDao.java (best guess)
### The error may involve rivermap.cn.wemapsyncserver.dao.UploadShapeDao.insertBatchProperty-Inline
### The error occurred while setting parameters
### SQL: insert into dt_shape_property_simple (shapeid, userid, groupid, create_userid, name, type, name_alias, stringvalue, doublevalue, intvalue, blobvalue, version, datasize, createtime, modifytime, deleted) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) , (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
### Cause: java.sql.SQLSyntaxErrorException: This version of ShardingSphere-Proxy doesn't yet support this SQL. 'You have an error in your SQL syntax'

### Reason analyze (If you can)
* Use the configuration mentioned in #17888 and add a `blobvalue` field of `blob` type in the `remark` table.

java
 public class RemarkPO {

 private byte[] blobValue;

 public byte[] getBlobValue() {
 return blobValue;
 }

 public void setBlobValue(byte[] blobValue) {
 this.blobValue = blobValue;
 }
}

* `insert(remark)`
* The execution conditions and results are as follows:
 * `remark.setBlobValue(null)`: OK
 * `remark.setBlobValue(new byte[5])`: OK
 * `remark.setBlobValue(new byte[0])`: throw SQLSyntaxErrorException

I found the SQL was not rewritten correctly.

[INFO ] 2022-05-25 15:44:55.387 [ShardingSphere-Command-9] ShardingSphere-SQL - Logic SQL: INSERT INTO remark(userid, content, `time`, b) VALUES (4, 'test0', '2022-05-25', _binary'blob')
[INFO ] 2022-05-25 15:44:55.387 [ShardingSphere-Command-9] ShardingSphere-SQL - SQLStatement: MySQLInsertStatement(setAssignment=Optional.empty, onDuplicateKeyColumns=Optional.empty)
[INFO ] 2022-05-25 15:44:55.387 [ShardingSphere-Command-9] ShardingSphere-SQL - Actual SQL: p1 ::: INSERT INTO remark(userid, content, `time`, b) VALUES (4, 'test0', '2022-05-25', 'binary'blob')


Question: [Feature] We need API version for endpoints?

This has been implement following the merge of #2080.

It will be difficult for us to do this. It would mean we would have to duplicate the API each time we decide to alter the products identifiers. @captn3m0 opened a discussion about that, let's continue there.

[Feature] We need API version for endpoints
Even we are now in alpha phase for our API , I still think we need a version in endpoints.
This will let us break api whenever we need to change structure without breaking userspace.

For example
right now we are using ` URL endpoint
my suggestion is adding version part in this endpoint such as
` URL `v1` `/{product}.json`

To keep it backward compatibility we can use redirection for a while
 URL -> URL

And whenever we need a break in api we just put new to v2 so v1 users still can continue to get values
with a warning that v1 is deprecated and will be removed

@captn3m0 @generalmimon @hebbet @BiNZGi @marcwrobel @adriens

There's a lot of maintenance burden maintaining old versions of the API, so we'll need some way to avoid that as well. I think just establishing a changelog + announcement process for any API changes should be good enough for now, with current URL redirects to `/v1/`.

Long-term:

- Alpha: API breaks whenever, no guarantees (We're here now)
- Beta: API schema is stable. We'll announce any changes, breaking or otherwise in advance. Breaking changes will be avoided, as much as possible. Field additions are still possible. Bug fixes are still allowed (such as fixing templates).
- Stable: Any breaking changes will be implemented via a separate version endpoint. Old endpoints will stay functional for [X] months, and default endpoint (without a version) will upgrade to new version after [Y] months.

Does this mean we'll need to add endoflife.date as a product on the site 😏 ?

Does it mean you'll have to maintain old APIs for retro-compat changes ?...

This has been implement following the merge of #2080.

Duplicate of #2331.

Add API endpoint containing its current version
Add API endpoint containing its current version

**Is your feature request related to a problem? Please describe.**
Currently, the API is not versioned.
This means that every change to its JSON structure, data types or data itself comes up as unexpected to the users.
In effect, this breaks applications using the API.

Examples of possibly app-breaking changes to the API:
 URL
 URL

**Describe the solution you'd like**
Add API endpoint, in a way that it can be accessed at:
 URL
and contains:
`{"version":" "}`

This would require support of semantic versioning of the API with major releases being JSON structure, data type changes etc. and minor releases being product / product details value updates.

**Describe alternatives you've considered**
None.

**Additional context**
This addition would prove endoflife.date more development-friendly.

Suggestion: use `schema_version` instead of version. Version has other meanings in the API, so it might be confusing.

I would like to keep versioning similar to URL which uses the following:

@filip-szweda Our api is still in alpha thus it might be a little unstable , thats the main reason we are trying to implement this one : URL

FYI the approach taken in #2080 is to version the API using the URL (e.g. `/api/v1`, `/api/v2`...) and incrementing the version when there is breaking changes, such as field removal.

This approach makes more sense in our particular context. It allows us to have multiple and concurrent versions of the API to give time to our users to migrate. This would not be possible with a `version` field because endoflife.date is just a static site.

We could also add a `version` (or `schema_version`) field in the response. Maybe it would be useful when an older version of the API is removed and old API URLs are redirected to the new API URLs ? What do you think @filip-szweda, @usta, @captn3m0 ?

It would also be helpful when we make minor schema changes (such as field additions).

I really like your approach in URL 👍

One thing I'm a bit concerned about are changes to the API that do not add / remove fields like e.g. URL
Do you think such changes could also increment the API version, perhaps only its minor part (i.e. /api/v1.1, /api/v1.2, /api/v1.3)?


Question: Merge NDArray inits with fill and random?

Reassigning to Lang as this requires compiler changes in how we import builtins. There's not much the library itself can do short of providing a smaller transitive closure set of builtins/prelude.

Merge NDArray inits with fill and random
To reduce the number of initializer overloads for ndarray we could take the `__inits__` that are identical other than having either random or fill and make initializers that default to filling with zeros, and do not allow for the fill to be nonzero if the random argument is true.

mojo
fn __init__( inout self, *shape: Int, random: Bool = False, order: String = "C" ) raises:

and

mojo
fn __init__( inout self, *shape: Int, fill: Scalar[dtype], order: String = "C" ) raises:

becomes

mojo
fn __init__( inout self, *shape: Int, fill: Scalar[dtype]=0, random:Bool=False, order: String = "C" ) raises:
 if random == True and fill !=0:
 raise Error("numojo/core/ndarray:NDArray: __init__(*shape, fill, random)): Error if random is true you cannot set a fill value")

This would turn 8 of our initializers into 4 thus reducing our number of initializers from 14 to 10.

Thoughts?

I would suggest that the `random` argument should be removed completely from this way of creating arrays as it is not obvious that this is uniform distribution between 0 and 1.
Creating random arrays should be done using `nm.core.random. `. This would make is explicit what kind of random is being created and allow for parameters to be set, like `nm.core.random.randn(mean=1.0, std=2.0)` or `nm.core.random.randint(min=10, max=100)` etc etc.
What do you think @MadAlex1997 ?

@sandstromviktor It would be better aligned with NumPy that way. Admittedly most of the data structure and initialization stuff was @forFudan and @shivasankarka, I have handled the math and numerics backend part more. Perhaps they would be able to give their reasoning for it being that way?

The randomly generated scalar/array part of NuMojo is currently pretty limited maybe we could put that on the list for V0.3 along with the many changes for the upgrade to Mojo 24.5.

@sandstromviktor I agree with that. Initially we added it to default constructors as mojo `random` module is quite limited. I am implementing some functions for random number generators right now. I will move these random array constructors to that.

@MadAlex1997 what other changes do you have in mind for the randomly generated arrays for NuMojo v0.3?

I have created a PR #98 with some basic changes for the `random` and `ndarray` module. If you all have any suggestions to improve the random module, please share here or in the PR. Thanks!

@sandstromviktor Your suggestion is nice.

@MadAlex1997 @shivasankarka, Another reason why we did not have a better `random` module was that the built-in `random` module is imported by default (which is actually a known issue URL This leaking polluted the naming space and caused importing `random` from `numojo` to fail in many cases.

With this issue fixed, we can improve the `random` module soon.

[BUG] Functions, traits, structs and aliases are leaking into builtins and are importable from anywhere
### Bug description

As the title says, importing anything without a leading underscore in "./stdlib/builtin/anything.mojo" insert it into the list of stuff that does not need to be imported globally.

The fix should be to make anything that's defined in URL available globally, but not more. And in this file, we will import all the things that should be auto-imported.

### Steps to reproduce

Consider this code:

mojo
def main():
 print(now()) # I want this to call time.now()

you might think that it will not work. Well, if someone, by mistake, imports `now()` into any of the builtin files, this will work.
e.g. Add `from time import now` in `stdlib/src/builtin/hash.mojo`, then run

bash
./stdlib/scripts/build-stdlib.sh && MODULAR_MOJO_NIGHTLY_IMPORT_PATH=./build mojo trying_stuff.mojo

It will happily compile the code and print

13050285013500

Imports are all messed up in the stdlib because of this. When doing refactorings, suddently from functions are not auto-imported anymore and sometime some functions are auto-imported while we don't want them to be.

We should auto-import everything defined in a single file, not a whole package. After all, there are not that many things that should be auto-imported in mojo.

### Suggestion of the fix process

Since changing the imports in the whole stdlib and the internal codebase might cause a huge diff and a lot of conflicts, I will suggest the following:

* we populate `stdlib/src/builtin/__init__.mojo` with the current list of auto-imported names (so too much names, so we don't have to fix everything at once)
* The compiler devs change the rule for auto-imported names. Auto-import everything in `stdlib/src/builtin/__init__.mojo` instead of everything in `stdlib/src/builtin/*.mojo`. Since we filled `stdlib/src/builtin/__init__.mojo` with everything auto-imported before, no other changes are necessary.
* We remove progressively names from `stdlib/src/builtin/__init__.mojo` that shouldn't be auto-imported. We can even do one name by one name to avoid too many conflicts or too many changes in one PR.

### System information

shell
ubuntu 22.04 in wsl2
modular 0.7.2 (d0adc668)
mojo 2024.5.323 (1d9276ea)


Question: async_task fails to run?

These warnings seem to be from slf4j 1.x. Spark 3.5.0 updated its slf4j dependencies to 2.x and removed the log4j implementation that slf4j 1.x delegates to, hence the warnings.

slf4j 2.x seems to be backwards compatible, so should be a matter of upgrading our own slf4j dependencies. Spark already brings in a suitable log4j implementation.

I linked a branch that already fixes some submodules but there's more work left.

async_task fails to run
Integration tests that rely on `async_task` started failing:

Traceback (most recent call last):
 File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/async_task.py", line 234, in main
 batch_jobs = get_batch_jobs(batch_job_id, user_id)
 File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/async_task.py", line 201, in get_batch_jobs
 jvm.org.slf4j.MDC.put(jvm.org.openeo.logging.JsonLayout.UserId(), user_id)
 File "/opt/venv/lib64/python3.8/site-packages/py4j/java_gateway.py", line 1664, in __getattr__
 raise Py4JError("{0} does not exist in the JVM".format(new_fqn))
py4j.protocol.Py4JError: org.slf4j.MDC does not exist in the JVM

Running from the command line shows more details:

`java.lang.NoSuchFieldError: mdc`

Probably has to do with the recent upgrade to Spark 3.5, which upgraded its log4j and slf4j dependencies.

Removing the Hadoop classpath fixed `async_task`, although this is in the logs:

SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See URL for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See URL for further details.

Fixing URL fixed `async_task` logging and got rid of those SLF4J warnings above.

Logging went missing when running tests
For example when running `testWriteMultibandTemporalRDDWithGapsSeparateAssetPerBand`, but probably is an issue with any test.
The logs show:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See URL for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See URL for further details.

And then many normal logs are not shown anymore.

Fixing the issue here might fix it also for `async_task`: URL

cc @bossie


Question: [FEATURE] Allow to disable SSL/TLS communication?

Inter node request carries user context (which contains sensitive information like user credential details, roles etc..), hence for securing user's sensitive information we need to enable node to node encryption.

So, it is not possible to disable SSL at transport layer.

Duplicate of URL

[FEATURE] Allow to disable SSL/TLS communication
**Is your feature request related to a problem?**
The `security` plugins comes with SSL/TLS communication out of the box. This certainly should be the default, but in certain environments which use other means of secure communication (for example, IPSec), it would be great to offer the option to disable SSL/TLS.

**What solution would you like?**
Offer the option to disable SSL/TLS.

**What alternatives have you considered?**
N/A

**Do you have any additional context?**
It might not be easy to do but preliminary discussion hinted it is feasible, in general.

[Triage] @peternied Could you please provide some context around the limitations of the securityadmin tool relying on TLS/SSL?

There is a functional concern that the security plugin depends on the admin certificate to authenticate a super admin user, without support for this there are many scenarios that are not possible. If the certificate was removed as a requirement for these APIs then this could be opened up.

Related issues
- URL
- URL
- URL

+1 for sure. To basically echo the sentiments of the original thread, #37, if you're already running all nodes and connections within a VPC, the SSL is redundant.

@reta With #2411 [1] merged, I think a large part of this issue is resolved - I am closing. If there are there other features that are not accessible, could you file issues on them or reopen with more details?

- [1] #2411

@peternied From what I understand, the original intent of this request is to disable Transport level TLS (instead of REST-level TLS which the security plugin already supports disabling)

At one point it looks like this feature may have been under development (See URL but I'm not sure of the status of that.

The major concern pointed out here is that disabling Transport TLS has consequences for the logic around nodes joining the cluster securely. A major feature of the security plugin is either the static `nodes_dn` setting in `opensearch.yml` or the dynamic `nodes_dn.yml` file which is part of the security index. In order for a node to join the cluster securely it must present a certificate that matches a known node_dn.

disable TLS for Transport, enable for Rest
Is it possible to disable encryption for node-to-node transport but have encryption for REST Layer ?

like to set _false_ for `opendistro_security.ssl.transport.enabled` (and comment/disable `opendistro_security.ssl.transport.*`),
and keep _true_ for `opendistro_security.ssl.http.enabled`

unfortunately I get
`Caused by: java.lang.IllegalStateException: opendistro_security.ssl.transport.enabled must be set to 'true'
`

if security enabled on both, everything works fine.
do I miss something?

thanks

I think you cannot disable Transport SSL as of now, i have opened a issue for the same
 URL

Possible Duplicate of #36

@hardik-k-shah , understood, but this depends on the design I'd say.
The transport can happen within a data dmz where the traffic is not exposed and the only way to get data in is ingest in front of it and coordinating nodes ,so in this case I'd need encryption on REST layer only.

ingest -> node1-node2-...nodeN -> coordinating node

so, if there is no chance to get this integrated it might be worth to remove that from the documentation and config examples.

I agree that this would be very convenient in topologies residing in DMZ.
+1

Hi @nean-and-i @JulienCarnec - Thanks for raising this use case. Can you provide details about scenarios that exist for you in a single DMZ. Currently transport SSL cannot be disabled. Keeping this as an action item to investigate further.

@alolita , we have ES cluster in a VPC, with Kibana behind a HTTPS Proxy, we use ODFE for Logs, Analytics and Alerting. We do not required HTTPS or Node-to-Node encryption.

Hi @alolita ,
In our use-case we use ODFE inside a secured Kubernetes cluster.

Hi @alolita ,
I'm using an EL cluster in one of our datacenters where I have dedicated ingest nodes for inserting data into the cluster and on the other hand I also have dedicated coordinator nodes for queries.
The data nodes are in a dedicated not exposed vlan, so only ingest and coordinator nodes are able to access the data nodes via a dedicated backend connection into the vlan.
So everything is separated and there is no chance to access/sniff the transport traffic.
This means there are reasonable examples/design/architecture where transport and rest is separated, hence it could make sense to separate encryption.
I was thinking of avoiding as much additional processing as possible, I want to go for as low latencies as possible on the cluster and saturating its 10G interlinks.

Another request for this here... We really only have a need for the authentication backends (LDAP, OIDC et al.) and no need for any encryption support.

There are really a few sets of different functionality in this plugin that ought to be individually available, (1) encryption (transport and REST layers), (2) authc and (3) authz.

This issue is even more pressing given that ES's official ECK K8s operator enforces the use of SSL via xpack. With this plugin also forcing its own SSL on the transport layer I imagine there are going to be conflicts, which may prevent the use of this plugin with the official ECK operator (although I am still poking around to find a workaround for this).

I am trying to use this plugin in a kubernetes cluster with a service mesh which is already taking care of inter-node secure communication and would also like to turn this off

Same here, I am running k8s istio service mesh with mTLS in STRICT mode. I would to turn TLS off.

Appreciate that you'd like to enforce sound security practises, but in the age of microservices... terminating TLS is not OFFE's job.

@hardik-k-shah what is the product management thoughts about this request. Is this planned to be implemented or there is no plan for this now?

I am also facing same issue. We are running a security k8s cluster and it is not that easy to generate correct certs for internode/interpod communication.
Removing the requirement to use TLS communication, but still keeping other security features, would be very much welcoming feature.

Ok, I can report some progress for those who want to use this security plugin but are currently unable due to various reasons, eg. related to certs and/or kubernetes.
Basically, the solution is to heavely adjust helm chart:
- remove data & master node and use just a client node which gets two additional roles (of being data and master) -> URL
- because we use traefik as ingress provider (its `ingressroute`), client's ingress has to be adjusted as well. -> URL
- becasue, your ES client becomes master and data node at the same time, you will need to ensure that you expose a correct set of services -> URL
- the strategy is then doing following: you expose ES on a ("public") ingress, e.g. `elastic.devops.firm.com` and do the same with kibana (with its security plugin) which points to that `httpS://elastic.devops.firm.com`. We run them both fine now, from 1 cluster, and ES is protected by default with password admin:admin which you can change. Same goes to kibana.

This is by no means optimal because in principle if you have just large cluster (we have just minikube) you dont need to expose ES via ingress. But doing so, puts ES on SSL and kibana is then quite easy for lift off.
Unfortunately, I am not able to post code here but that should help some people, until this issue will get higher priority in the roadmap. 💯 (which i hope it will)

@dmpe, this issue is not related to helm charts or K8s. It requests an option to disable TLS at the transport layer in Opendistro. It is not related to Traefik, Helm deployments, Minikube, or other aspects of a K8s setup.

Can you please edit your comment to make it clear that it does not resolve the issue, or (better!) delete your comment here and create a new issue describing your workaround?

While it may help some users, I am concerned your comment implies that there is a workaround. This might lead to premature closure of the issue (if the maintainers actually checked their Github issues, which I suspect they do not, but still...)

Happy to provide you some advice on your setup for if you want to dm me @MilesGarnsey on Twitter.

Sorry for the delay!
We will bring this to our next prioritization meeting and update here soon.

hey @hardik-k-shah - do you have an ETA of when the update of when we would be able to turn off inter-node TLS through elasticsearch.yml?

Hello all,

Yes, we have discussed this issue internally and decided that for benefit of ODFE community, we will be supporting configuration to disable TLS. (Though we strongly recommend not to disable TLS).

And we are currently testing TLS disabling feature.
Same time we are also working on introducing Dual Mode (so moving from TLS disable to enable will become simpler without down time. )

Thank you all.

@hardik-k-shah will it be available in next release ? 1.10.1 ? thanks

I've brought this issue up in the meetup couple of weeks ago but didn't get any response yet.
I do not see this into the current roadmap either. Is this something that is being worked on, or not yet?

Thanks a lot in advance for any info.

@spapadop Sorry we missed this one from the community meeting a while ago but I'm checking into it now.

@spapadop Not sure, to be honest. I know the team has been juggled a bit. Let me find out. Maybe @davidlago knows about the mysterious roadmap item?


Question: "ValueError: value.index does not match parent’s axis 0 names" error when trying to read h5ad that was processed through scVI?

I'm not the biggest fan of it either, but it's hard to move away from. Ideally there would be entirely separate methods for positional indexing and label based indexing (i.e. `loc` and `iloc`) but that isn't our API. When I first came across this, implementing that kind of API would have been very difficult, though could be easier now. For backwards compatibility we could probably never support integer label based access with `adata[idx]` syntax. There are also a lot of assumptions built into the ecosystem about this.

-------------------------------

python
adata = sc.AnnData(data)
adata.obsm['X'] = data

@flying-sheep, if the passed object has a numeric index, we could try converting it to strings and check if that matches?

----------------------

@dburkhardt, there are some previous issues on this topic you might be interested in:

* URL
* URL

"ValueError: value.index does not match parent’s axis 0 names" error when trying to read h5ad that was processed through scVI
Hi !

I am new to the Scanpy/Anndata ecosystem, and trying to use scVI for data integration.

I trained a scVI model on an Anndata object, saved the object as h5ad, and now I get an error when trying to read the file.

The error occurs when using scanpy.read_h5ad or scvi.data.read_h5ad, and I found someone else reporting the error in this repo before , so I am posting the issue here. Kindly let me know if I should post this somewhere else.

py
adipo_all = scvi.data.read_h5ad("/home/yyyyy/analysis/anndata_working/adipo_sn_01112021_trained_v1.h5ad")

pytb
/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py:120: ImplicitModificationWarning: Transforming to str index.
 warnings.warn("Transforming to str index.", ImplicitModificationWarning)

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/scratch/33545574/ipykernel_5065/2292619582.py in
----> 1 adipo_all = scvi.data.read_h5ad("/home/yyyyy/analysis/anndata_working/adipo_sn_01112021_trained_v1.h5ad")

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_io/h5ad.py in read_h5ad(filename, backed, as_sparse, as_sparse_fmt, chunk_size)
 435 _clean_uns(d) # backwards compat
 436
--> 437 return AnnData(**d)
 438
 439

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py in __init__(self, X, obs, var, uns, obsm, varm, layers, raw, dtype, shape, filename, filemode, asview, obsp, varp, oidx, vidx)
 320 varp=varp,
 321 filename=filename,
--> 322 filemode=filemode,
 323 )
 324

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py in _init_as_actual(self, X, obs, var, uns, obsm, varm, varp, obsp, raw, layers, dtype, shape, filename, filemode)
 508
 509 # TODO: Think about consequences of making obsm a group in hdf
--> 510 self._obsm = AxisArrays(self, 0, vals=convert_to_dict(obsm))
 511 self._varm = AxisArrays(self, 1, vals=convert_to_dict(varm))
 512

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in __init__(self, parent, axis, vals)
 233 self._data = dict()
 234 if vals is not None:
--> 235 self.update(vals)
 236
 237

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/_collections_abc.py in update(*args, **kwds)
 Supraclavicular 839 if isinstance(other, Mapping):
 840 for key in other:
--> 841 self[key] = other[key]
 842 elif hasattr(other, "keys"):
 843 for key in other.keys():

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in __setitem__(self, key, value)
 149
 150 def __setitem__(self, key: str, value: V):
--> 151 value = self._validate_value(value, key)
 152 self._data[key] = value
 153

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in _validate_value(self, val, key)
 211 # Could probably also re-order index if it’s contained
 212 raise ValueError(
--> 213 f"value.index does not match parent’s axis {self.axes[0]} names"
 214 )
 215 return super()._validate_value(val, key)

ValueError: value.index does not match parent’s axis 0 names

This is the offending anndata object.

AnnData object with n_obs × n_vars = 123472 × 36795
 obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_spliced', 'nFeature_spliced', 'nCount_unspliced', 'nFeature_unspliced', 'nCount_HTO', 'nFeature_HTO', 'HTO_maxID', 'HTO_secondID', 'HTO_margin', 'HTO_classification', 'HTO_classification.global', 'hash.ID', 'nCount_SCT', 'nFeature_SCT', 'batch', 'n_genes', 'annot', 'sample_origin', 'day', 'depot', 'tissue', 'timepoint', 'n_genes_by_counts', 'total_counts', 'total_counts_percent_ribo', 'pct_counts_percent_ribo', '_scvi_batch', '_scvi_labels'
 var: 'features', 'spliced_features', 'unspliced_features', 'n_cells', 'highly_variable', 'highly_variable_rank', 'means', 'variances', 'variances_norm', 'highly_variable_nbatches', 'percent_ribo', 'n_cells_by_counts', 'mean_counts', 'pct_dropout_by_counts', 'total_counts'
 uns: 'hvg', 'sample_origin_colors', 'depot_colors', '_scvi'
 obsm: '_scvi_extra_categoricals', '_scvi_extra_continuous', 'X_scVI'
 layers: 'counts', 'spliced', 'unspliced'

Whereas the previous version below can be read just fine. scanpy.pp.highly_variable_genes and sc.pp.calculate_qc_metrics were run followed by model training in scVI to get the above anndata.

AnnData object with n_obs × n_vars = 123472 × 36795
 obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_spliced', 'nFeature_spliced', 'nCount_unspliced', 'nFeature_unspliced', 'nCount_HTO', 'nFeature_HTO', 'HTO_maxID', 'HTO_secondID', 'HTO_margin', 'HTO_classification', 'HTO_classification.global', 'hash.ID', 'nCount_SCT', 'nFeature_SCT', 'batch', 'n_genes', 'annot', 'sample_origin', 'day', 'depot', 'tissue', 'timepoint'
 var: 'features', 'spliced_features', 'unspliced_features', 'n_cells'
 layers: 'counts', 'spliced', 'unspliced'

**Libraries: scanpy==1.8.1 anndata==0.7.6 umap==0.5.1 numpy==1.20.3 scipy==1.7.1 pandas==1.3.3 scikit-learn==0.24.2 statsmodels==0.13.0rc0 python-igraph==0.9.6 pynndescent==0.5.4**

I had a similar error and doing the following resolved the error. Make sure all 'obsm' objects that are dataframes have the same index as the 'obs' dataframe.

Seems like this is solved, and the discussion happens in #311

Please tell us if you need anything

Is logging "Transforming to str index." necessary?
I do a lot of moving back and forth between DataFrames and AnnData objects, and the "Transforming to str index." warning can get annoying, especially if I'm running things in a loop:

<img width="1052" alt="image" src=" URL

In these cases, I'm creating a DataFrame using something like:

python
data = np.random.normal(size=(1000,100))
data = pd.DataFrame(data)
... # code that expects data as a DataFrame
adata = sc.AnnData(data)

Does this need to be logged in a manner that is printed by default? Is there an easy way to disable this?

It looks like this is the offending line
 URL

It was introduced in 34f4eb63710628fbc15e7050e5efcac1d7806062

It triggers even if we have a RangeIndex, which shouldn’t trigger a warning.

I’m not sure if a Int64Index should trigger a warning, as it’s easy to get one by slicing a DataFrame with a RangeIndex.

I also realized this bug raises an index error when you try to add `obs` annotations using a rangeindex:

python
import scanpy as sc
import pandas as pd
import numpy as np
data = pd.DataFrame(np.random.normal(size=(100,2)))

adata = sc.AnnData(data)
adata.obsm['X'] = data
Transforming to str index.
Transforming to str index.

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
 in
 5
 6 adata = sc.AnnData(data)
----> 7 adata.obsm['X'] = data

~/.local/lib/python3.8/site-packages/anndata/_core/aligned_mapping.py in __setitem__(self, key, value)
 148
 149 def __setitem__(self, key: str, value: V):
--> 150 value = self._validate_value(value, key)
 151 self._data[key] = value
 152

~/.local/lib/python3.8/site-packages/anndata/_core/aligned_mapping.py in _validate_value(self, val, key)
 209 ):
 210 # Could probably also re-order index if it’s contained
--> 211 raise ValueError(
 212 f"value.index does not match parent’s axis {self.axes[0]} names"
 213 )

ValueError: value.index does not match parent’s axis 0 names

What's the reason for changing the index to str in the first place? Why not use the pd.Index object passed when you create the AnnData object?

AnnData objects use positional indexing when you supply numeric indices, but access by name when pass strings. If we allowed numeric indices that would statements like: `adata[0]` ambiguous. There's a little more detail on this in the AnnData docstring notes.

@flying-sheep, this is a logging warning, not an actual `warnings.Warning` right? It should probably be an actual warning so it's easier to catch and silence.

Default behavior is that warnings display once per location where it’s raised: URL

does that mean it gets shown once per location where it’s raised? that would be bad, as we want it to show every time a new AnnData object is created or has its index replaced.

Hmm. I'm not quite sure. I had thought location meant context/ stack-level, but I'm not sure if that's true. It'll definitely warn less than @dburkhardt is reporting:

python
In [1]: import warnings

In [2]: def dangerous():
 ...: warnings.warn("Woah there!", UserWarning)
 ...:

In [3]: def foo():
 ...: for i in range(10):
 ...: dangerous()
 ...:

In [4]: foo()
/usr/local/bin/ipython:2: UserWarning: Woah there!
 # -*- coding: utf-8 -*-

In [5]: foo()
/usr/local/bin/ipython:2: UserWarning: Woah there!
 # -*- coding: utf-8 -*-

Thanks for the help here all! I don't think this issue is closed though as the following code will still raise an error, when I don't think it should:

python
import scanpy as sc
import pandas as pd
import numpy as np
data = pd.DataFrame(np.random.normal(size=(100,2)))

adata = sc.AnnData(data)
adata.obsm['X'] = data

Thanks for the clarification on positional vs names indices @ivirshup. I'm not sure that I like this paradigm of assuming that there need be `str` labels for each column. I understand that in the context of scRNA-seq one will often have cell barcodes, but it was my impression that `AnnData` is meant to be a more generalized data storage class (hence little mention of any specific technology in the docs). I usually think of it as `pd.DataFrame`-plus.

I often have data with only numeric positional indices. It would be nice to have full support for this use-case correct? Shouldn't that code above work properly without raising an error? I think to do that you might need to let users who don't have non-numeric `obs_names` or `var_names`.

Another consideration here is what to do if I were to select a subset of that DataFrame before creating the AnnData object:

python
adata = sc.AnnData(data.iloc[10:])

then the `obs` at position `0` will have index `10` in the DataFrame.

It does? I thought we test if a DataFrame has a RangedIndex and don’t warn if it’s the case.

that would work I guess. not really pretty, but I can’t think of how it would have unintentional side effects

Let’s track this in #31. #954 fixed a bit more issues with strings.


Question: Ren'Py Launcher Not Responding?

I'm closing this, as it's likely to be some other process that's causing this problem.

Ren'Py Launcher Not Responding
Hello!

I recently got back into Renpy. I downloaded the latest build, Renpy 8.3.2. It worked fine at first, but now, for some reason, the launcher will not work. It instantly goes into "not responding." I have tried reinstalling and installing it into a different folder path. Nothing has worked.

This issue started a bit after installing VS Code and its extensions for Renpy and Python, such as Pylance. However, I am not positive if its directly responsible for the issue, since after uninstalling it the issue stayed.

The log file says everything loaded fine, there is no traceback file.

Any help would be much appreciated!

Did you install Visual Code via the Launcher or manually? The reason I ask is because the Launcher allows you to download Visual Code with the current version of Renpy Extension (which had been changed recently). If you did not do it through the Launcher:

Try uninstalling Visual Code completely, restarting, and loading the Launcher again. If it works after that point, try going to Preferences in the Launcher, then Text Editor, and click on Visual Code. It will download and install both Visual Code and the current version of the Renpy extension.

Another thing you can try if uninstalling Visual Code does not fix it, try on another User account for your system. User Profiles (especially Windows) can cause strange behaviors if something happens (yes, that is vague, but there is a lot of things that occurs with a User Profile). If that fails and you have access to another system, check to see if the same behavior happens on that.

You can also try:

 URL

Select a build from the Ren'Py 8 Fix column. These are nightly fixes that fix issues found in Renpy that might not be in 8.3.2 whereas Renpy 8 Development meant for developing the next version (in this case, 8.4) of Ren'Py. This way fixes could be released sooner without waiting for Renpy 8.4 to come out.:)

This sounds like URL - did you per chance move the Launcher's window to a secondary monitor, and close it there the last time that it was working properly?

If so, try creating a new .rpy file in your `renpy-8.3.2-sdk/launcher/game/` folder, and write the following inside it:

py
init 999 python:
 preferences.restore_window_position = False

See if that indeed fixes the issue.

Fixed it by adding a new file in renpy-8.3.2-sdk/launcher/game/

named the file fix.rpy, and adding the following code

init 999 python:
 preferences.restore_window_position = False

Game freezes when restoring window position from secondary monitors
Encountering an issue in Ren'Py 8.2.0 where, if I launch a game from the Launcher, move the game's window to a secondary monitor, and close the game there, on the next launch the game freezes on a fully white screen and becomes non-responsive until it is needed to be forcefully killed. This does not happen if the window was only moved elsewhere on the Primary monitor - that works as intended and the previous position gets restored.

The Primary monitor is a 21:9 2560x1080p, and the secondary monitors are a 16:9 4K (3840x2160px) one and a 16:10 1920x1200px; this happens when I move the game's window to either of them.

I was initially wondering if this was due to the varying resolutions of the monitors, but that did not seem to be the case: I asked a few people with multi-monitor setups to test it out, one whose both monitors had the same resolution, and two others with also varying resolutions ((1920x1080px + 1920x1080px + 4K), and (2560x1080p+ 1920x1080px)); and they did _not_ experience this issue.

I also ran some experiments with making each monitor the Primary:
- If the 16:10 one is the Primary, the results are the same; moving the window to the other monitors makes it freeze on the next launch.
- If the 4K one is the Primary, moving the window to other monitors has no special result: it always gets reset back to the center of the 4K one, with no freezes or anything. It does also restore the position if it was set elsewhere on the 4K monitor itself.

I also tried changing the resolutions of all 3 monitors to 1920x1080px via Windows settings, that did not make a difference.
Also tried unplugging each one of the monitors entirely, and the results were the same with the remaining monitors in all combinations.

Unsure of what else to test, but I tried printing out `preferences.window_position` and `preferences.window_position_layout` during Init to maybe help provide some clue:

py
# Moved from Primary 21:9 to elsewhere on Primary 21:9:
renpy.game.preferences.window_position == (649, 31)
renpy.game.preferences.window_position_layout == ((0, 0, 2560, 1080), (-3840, -465, 2560, 1440), (2560, 2, 1920, 1200))

# Moved from Primary 21:9 to secondary 16:9 4K (freeze):
renpy.game.preferences.window_position == (-3560, -265)
renpy.game.preferences.window_position_layout == ((0, 0, 2560, 1080), (-3840, -465, 2560, 1440), (2560, 2, 1920, 1200))

# Moved from Primary 21:9 to secondary 16:10 1920x1200px (freeze):
renpy.game.preferences.window_position == (2633, 74)
renpy.game.preferences.window_position_layout == ((0, 0, 2560, 1080), (-3840, -465, 2560, 1440), (2560, 2, 1920, 1200))

Log file: log.txt

This is on Ren'Py 8.2.0,
NVIDIA driver version: 551.23 (latest at this time)
OS:
 Edition: Windows 10 Pro
 Version: 22H2
 OS build: 19045.3930
 Experience: Windows Feature Experience Pack 1000.19053.1000.0

Screenshot 2024-01-29_
Screenshot 2024-01-29__

I don't know what the problem is, but this is very unlikely to be a Ren'Py problem. Other than positioning the window where you last place it, Ren'Py is working the same way regardless of monitor. I'd suggest reporting this issue to Nvidia to see if they can help as all.

@juniorthejunior Have you tried completely uninstalling the Nvidia Drivers, downloading the latest from Nvidia, restarting Windows, and Reinstalling the Drivers? Going into Device Manager, right-click on the video card(s), click on uninstall, and click the checkbox that Asks if you want to Delete Drivers. It will probably be something Nvidia will ask to see if it is a problem with their drivers, or if it is a problem with your currently installed drivers.

One thing about "updating drivers", it may not ALWAYS put the newest version of everything. There could be a lingering issue that could cause odd behaviors.

Additionally, they also may ask you if this happens in another profile (Username). Corrupted (or starting to corrupt) profiles can cause wide range of weird issues. It will help both them and you determine if it is system-wide or just account-wide.

Just two things to check that I did not see in your lists because I do not know how quick Nvidia is with responding to support requests.

Thanks for the suggestions. Followed the steps - deleted via Device Manager, restarted, installed 551.23 (the latest), restarted - and the problem still persists. I also repeated the process except uninstalling the driver using DDU (a third-party "deep uninstaller" of sorts), and that made no difference either.

Also made a new Windows account/profile, downloaded and ran Ren'Py 8.2.0 and the results were the same there as well.

I'll have to put together a report to NVIDIA at some point in the future, though at least for the time being, I am disabling this feature entirely in my project, since I personally would not want a player to encounter it and have them have to deal with it (however rare it might be; if it's happened to me, it'll happen to someone else), especially for a mostly QoL feature.

I have almost the same issue. Found that it was Discord that was messing something up sometimes and that was causing the videos on YouTube to not play and Ren'Py games freezing on closing.


Question: What does this error mean and how do I resolve it?

When I played around with your equations, I was able to reproduce one of the exceptions.

Could it be that you were trying to simplify the equation for C8 which has 17 input variables? Setting the memory 2GByte is not enough for that, even with 12GByte the equation can't be simplified with QMC.

The root cause of the error messages is a Java heap space error.

What does this error mean and how do I resolve it?
Hello,

I am getting this error when running an analysis:

Error during simplification
Too many variables used in Z^{n+1},
12 are allowed variables but 19 were found.

I haven't found any information on this issue and don't know how to resolve it.

I've attached the file. I am trying to use this to upload to an ATF1504 CPLD.

CPLD_RAM_2MB.zip

Thanks!

The simulator uses QMC (Quine and McCluskey) to simplify logic equations, and the problem with QMC is that while it always provides an exact minimum solution, the memory consumption and computing time increases exponentially with the number of input variables. Therefore QMC is not suitable for functions with many inputs. To avoid that it takes too long until the result is available, the calculation is not performed if there are more than 12 inputs.
With such a large number of inputs you would have to use e.g. the heuristic Espresso algorithm, which is faster and uses less memory, but does not always provide the minimum solution.
Unfortunately, I have not yet got around to implementing the Espresso algorithm.

see also #495

Is there a way to increase the number of inputs at your own risk?

Yes! You can start the simulator with the command line argument `experimental`.
This avoids the error message. But in this case you should also increase the allowed max memory consumption.

java -Xmx8192m -jar Digital.jar experimental

I terminated the program after about 4 hours. Memory consumption was up over 10 GB. I really like the program but will have to use it for logic simulations and smaller CPLD's only.

As I said before: Exact solution algorithms are of order O(2^n). These are not suitable for minimizing equations with more than 12-14 variables. And this is sufficient for the application for which the simulator is intended (teaching).
But without a heuristic, non-exact solution method like e.g. the Espresso algorithm, it is not possible to find a solution in many cases.

Unable to build circuit with 17 inputs out of NAND gates
I wanted to see how many NAND gates my 8 bit carry-lookahead adder would have if it was built solely using them.
The adder was created from 32 boolean expressions, later each circuit that depended on another connected with tunnels (some expressions' variables were results from other expressions). These are the 32 expressions:

let p0 = a0^b0;
let p1 = a1^b1;
let p2 = a2^b2;
let p3 = a3^b3;
let p4 = a4^b4;
let p5 = a5^b5;
let p6 = a6^b6;
let p7 = a7^b7;
let g0 = a0 b0;
let g1 = a1 b1;
let g2 = a2 b2;
let g3 = a3 b3;
let g4 = a4 b4;
let g5 = a5 b5;
let g6 = a6 b6;
let g7 = a7 b7;
let S0 = p0 ^ C0;
let C1 = g0 + p0 C0;
let S1 = p1 ^ C1;
let C2 = g1 + p1 g0 + p1 p0 C0;
let S2 = p2 ^ C2;
let C3 = g2 + p2 g1 + p2 p1 g0 + p2 p1 p0 C0;
let S3 = p3 ^ C3;
let C4 = g3 + p3 g2 + p3 p2 g1 + p3 p2 p1 g0 + p3 p2 p1 p0 C0;
let S4 = p4 ^ C4;
let C5 = g4 + p4 g3 + p4 p3 g2 + p4 p3 p2 g1 + p4 p3 p2 p1 g0 + p4 p3 p2 p1 p0 C0;
let S5 = p5 ^ C5;
let C6 = g5 + p5 g4 + p5 p4 g3 + p5 p4 p3 g2 + p5 p4 p3 p2 g1 + p5 p4 p3 p2 p1 g0 + p5 p4 p3 p2 p1 p0 C0;
let S6 = p6 ^ C6;
let C7 = g6 + p6 g5 + p6 p5 g4 + p6 p5 p4 g3 + p6 p5 p4 p3 g2 + p6 p5 p4 p3 p2 g1 + p6 p5 p4 p3 p2 p1 g0 + p6 p5 p4 p3 p2 p1 p0 C0;
let S7 = p7 ^ C7;
let C8 = g7 + p7 g6 + p7 p6 g5 + p7 p6 p5 g4 + p7 p6 p5 p4 g3 + p7 p6 p5 p4 p3 g2 + p7 p6 p5 p4 p3 p2 g1 + p7 p6 p5 p4 p3 p2 p1 g0 + p7 p6 p5 p4 p3 p2 p1 p0 C0

I cloned the repository and with Digital built v0.24-137-ga08496b (as of a08496b5b132a49093a7a93a0bf4007ef2d7687a), running it with OpenJDK Runtime Environment (build 14.0.2+12) using `java -Xms2048M -jar Digital.jar experimental` and `java -Xms2560M -jar Digital.jar experimental`, both instances were able to generate identical tables. The sha256sum of the two saved tables is `37413a273d8bec7b51d4ba36ccd3c28903d1eb9a20866683251d02f6a132777e` and `diff` found no differences either. But the tables were generated with an error in both cases:

de.neemann.digital.analyse.expression.ExpressionException: ExpressionListenerStore not closed
 at de.neemann.digital.gui.components.table.ExpressionListenerStore.replayTo(ExpressionListenerStore.java:62)
 at de.neemann.digital.gui.components.table.ExpressionCreator.create(ExpressionCreator.java:93)
 at de.neemann.digital.gui.components.table.TableDialog.lambda$calculateExpressions$9(TableDialog.java:723)
 at java.base/java.lang.Thread.run(Thread.java:832)

With the table, I tried to build a circuit solely with NAND gates, and the following error issued:

de.neemann.digital.analyse.expression.ExpressionException: Não há equações mínimas!
 at de.neemann.digital.gui.components.table.BuilderExpressionCreator.create(BuilderExpressionCreator.java:50)
 at de.neemann.digital.gui.components.table.TableDialog.createCircuit(TableDialog.java:642)
 at de.neemann.digital.gui.components.table.TableDialog.createCircuit(TableDialog.java:630)
 at de.neemann.digital.gui.components.table.TableDialog.access$1500(TableDialog.java:75)
 at de.neemann.digital.gui.components.table.TableDialog$26.actionPerformed(TableDialog.java:554)
 at java.desktop/javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967)
 at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308)
 at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405)
 at java.desktop/javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262)
 at java.desktop/javax.swing.AbstractButton.doClick(AbstractButton.java:369)
 at java.desktop/javax.swing.plaf.basic.BasicMenuItemUI.doClick(BasicMenuItemUI.java:1020)
 at java.desktop/javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(BasicMenuItemUI.java:1064)
 at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297)
 at java.desktop/java.awt.Component.processMouseEvent(Component.java:6636)
 at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3342)
 at java.desktop/java.awt.Component.processEvent(Component.java:6401)
 at java.desktop/java.awt.Container.processEvent(Container.java:2263)
 at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5012)
 at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321)
 at java.desktop/java.awt.Component.dispatchEvent(Component.java:4844)
 at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4919)
 at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4548)
 at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4489)
 at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307)
 at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2764)
 at java.desktop/java.awt.Component.dispatchEvent(Component.java:4844)
 at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:772)
 at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
 at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715)
 at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
 at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
 at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95)
 at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:745)
 at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:743)
 at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
 at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
 at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:742)
 at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
 at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
 at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
 at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
 at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
 at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

If it is needed to know anything else, I'm glad to inform.

I'm not sure I understood the problem.
I have tested your input to the circuit generator, and this seems to work fine.

If you then try to create a truth table, it's is not possible because of not unique variables used.

But if you fix the internal used local variables (which should be done by the circuit generator automatically, i guess) there are only 17 inputs remaining (a0-a7, b0-b7 and c0) and you will get the truth table of an 8-bit adder which is the same for any 8-bit adder and therefore useless in respect of the carry-look-ahead logic.

You understood correctly the problem, but now I see how foolish on my part it was to think that whatever circuit I could build from the truth table, be it made out of two-input NAND gates or anything else, would represent the carry-lookahead circuit.

Not directly, but certainly. After I generated the circuit with the equations and I fixed the internal variables, just as you did, I hit "Analysis" from the "Analysis" menu and waited until the table was generated, when the errors occurred.

I think you are right with it being lack of memory, specially for C8. Using Digital as of 98dc38540bcd649fcd2b4ffe2089d2458e841298 and with the same parameters as before, I tried "Synthesis" of the "Analysis" menu, I opened the saved table and, after a while, this appeared: "java.lang.OutOfMemoryError: Java heap space". For context, this is the full output in the terminal:

[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - start optimization
[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - optimization finished
[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - start optimization
[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - optimization finished
[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - start optimization
[AWT-EventQueue-0] INFO de.neemann.digital.gui.components.table.TableDialog - optimization finished
de.neemann.digital.analyse.expression.ExpressionException: ExpressionListenerStore not closed
 at de.neemann.digital.gui.components.table.ExpressionListenerStore.replayTo(ExpressionListenerStore.java:62)
 at de.neemann.digital.gui.components.table.ExpressionCreator.create(ExpressionCreator.java:93)
 at de.neemann.digital.gui.components.table.TableDialog.lambda$calculateExpressions$9(TableDialog.java:723)
 at java.base/java.lang.Thread.run(Thread.java:832)
jul. 25, 2020 3:15:56 PM sun.awt.X11.XToolkit processException
WARNING: Exception on Toolkit thread
java.lang.OutOfMemoryError: Java heap space

de.neemann.digital.analyse.expression.ExpressionException: ExpressionListenerStore not closed
 at de.neemann.digital.gui.components.table.ExpressionListenerStore.replayTo(ExpressionListenerStore.java:62)
 at de.neemann.digital.gui.components.table.ExpressionCreator.create(ExpressionCreator.java:93)
 at de.neemann.digital.gui.components.table.TableDialog.lambda$calculateExpressions$9(TableDialog.java:723)
 at java.base/java.lang.Thread.run(Thread.java:832)

Indeed, by removed all outputs except C8 from the table through the option "Reorder/Delete Output Columns" in the "Edit" menu, the same error appears after a while: `java.lang.OutOfMemoryError: Java heap space`.

I think we may close the issue then.

The problem with QMC (Quine and McCluskey) is that while it always provides an exact minimum solution, the memory consumption and computing time increases exponentially with the number of input variables. Therefore QMC is not suitable for functions with many inputs. In such a case you would have to use e.g. the heuristic Espresso algorithm, which is faster and uses less memory, but does not always provide the minimum solution.
Unfortunately, I have not yet got around to implementing the Espresso algorithm.


Question: Invalid yaml?

@henryobiaraije This option was deprecated couple uears ago with Improvements we made with import flow to handle larger specifications better.

Could you explain the need to use this option and any issues that you might be running into?

Invalid yaml
- `/income/verification/documents/download`
- `/beta/credit/v1/bank_employment/get`
- `/deposit_switch/get`

All have API errors in their yaml

most can be fixed with indentation; the beta one has `x-hidden-from-docs: true` outside of a method, which means some tooling treats it as a method and crashes.

We validate the OpenAPI file using the validator at URL and it passes that validation. If there is a specific different validation tool that you are running or specific places where you think the indentation should be changed, please let us know.

Okay so for `/deposit_switch/get` and `/income/verification/documents/download` would you say that parameters is correctly indented, or indeed necesarry?

Even within the same file, the indentation is different to all the other endpoints featuring `parameters: []`

`/beta/credit/v1/bank_employment/get` has an `x-hidden-from-docs: true` which is not within a http method under a path. Is that correct?

I don't doubt you have automation. These endpoints are documented, but don't seem to work with URL when passed through URL being told to use examples.

Also when I try to use these endpoints using swagger generator I get errors like the following

{
 "message": "unknown format \"datetime\" is used in schema at path \"#/properties/initiated_date\""
}

which is a http 500 error.

Other tooling used (problematically) are things like postman

 "
 },
 "client_id": {
 "value": " "
 },
 "secret": {
 "value": " "
 }
}

which is what happens if I try to import ` URL (currently `master`)

None of this is intended to be problematic. When I asked about OpenAPI2 support, I accepted that OAS3 is what we should work with. I Also accept this is difficult and new and a hard area and that PR's are not really worth it, as they then need to be translated to some program that generates that file. I Just want things to work and not cause issues. Using swagger generator and verifying calls to plaid via postman are two ways of troubleshooting that implementations in code match the API-spec contract as well as making calls direct to plaid to gain visibility.

Thanks, that is super helpful detail.

The `x-hidden-from-docs: true` I can fix (it doesn't even do anything, so we can delete that line)
The `parameters` indentation is also an easy fix (and again, I'm not sure it's even doing anything)
The `format: datetime` thing is something we already caught internally and it's fixed on master, so that should be rolled out with next month's release.

The Postman error seems like unfortunately a limitation in Postman rather than anything wrong with the API specification, so I don't think there's anything we can do about that. There are some suggested workarounds floating around online, e.g.: URL

Super helpful thanks. The workaround I'm picking out from that little thread might be

openapi2postmanv2 -s root.yaml -o collection.json -p -O folderStrategy=Tags,includeAuthInfoInExample=false,optimizeConversion=false,stackLimit=50

I've applied the fixes locally and just did a find replace for the hyphenation of date-time; It's more that I'd sooner not write automated tools; it's not just plaid, every vendor has... insert quirky thing 👍

Error: Too many levels of nesting to fake this schema
### Is there an existing issue for this?

- [X] I have searched the tracker for existing similar issues and I know that duplicates will be closed

### Describe the Issue

When I use nesting in my OpenAPI definition and generate collection. The generated examples where nesting has been used show the following error:

Sample generated Example:

{
 "meta": {
 "page_size": 50,
 "continuation_token": "A Very Long JSON String"
 },
 "data": [
 {
 "service": {
 "id": "bc73d72d-3cca-433b-8229-ab8fd0f7c2f1",
 "name": "Software Development"
 },
 "posted_by": {
 "id": "05072d22-ab35-47ce-8d58-4ffe0d6fcda7",
 "name": "John Wick",
 "city": "Toronto",
 "country": "Canada",
 "phone": "+1 ***-***-3006",
 "phone_verified": true,
 "email": "us***@****.com",
 "email_verified": false,
 "posted_on": "2017-07-21T17:32:28Z"
 },
 "id": "05072d22-ab35-47ce-8d58-4ffe0d6fcda7",
 "lead_status": "pending",
 "credits": 22,
 "experts_responded": 3,
 "service_questions_and_answers": [
 {
 "question_id": {
 "value": " "
 },
 "question": {
 "value": " "
 },
 "answer": {
 "value": " "
 }
 },
 {
 "question_id": {
 "value": " "
 },
 "question": {
 "value": " "
 },
 "answer": {
 "value": " "
 }
 }
 ]
 },
 {
 "service": {
 "id": "bc73d72d-3cca-433b-8229-ab8fd0f7c2f1",
 "name": "Software Development"
 },
 "posted_by": {
 "id": "05072d22-ab35-47ce-8d58-4ffe0d6fcda7",
 "name": "John Wick",
 "city": "Toronto",
 "country": "Canada",
 "phone": "+1 ***-***-3006",
 "phone_verified": true,
 "email": "us***@****.com",
 "email_verified": false,
 "posted_on": "2017-07-21T17:32:28Z"
 },
 "id": "05072d22-ab35-47ce-8d58-4ffe0d6fcda7",
 "lead_status": "pending",
 "credits": 22,
 "experts_responded": 3,
 "service_questions_and_answers": [
 {
 "question_id": {
 "value": " "
 },
 "question": {
 "value": " "
 },
 "answer": {
 "value": " "
 }
 },
 {
 "question_id": {
 "value": " "
 },
 "question": {
 "value": " "
 },
 "answer": {
 "value": " "
 }
 }
 ]
 }
 ]
}

### Steps To Reproduce

1. Import attached API Spec
2. Generate Collection
3. View generated successful response example under Leads --> [admin,client,expert] Returns lead list

### Screenshots or Videos

image
Spec.txt

### Operating System

Windows

### Postman Version

9.18.2

### Postman Platform

Postman App

### Additional Context?

_No response_

Hi @umnex , thanks for reporting. I'm not able to reproduce using your example when importing when using path or tags folder organization. It looks at first glance like the 10 stack limit nesting level hasn't been reached either.
<img width="1779" alt="Screen Shot 2022-05-19 at 11 39 12 AM" src=" URL

Is there a different example you're using that has more nesting happening? To get around these limits, I'd recommend checking out the openapi-to-postman project and this thread in particular where these limits are discussed for how to bypass them:
 URL

Sorry I attached the spec where i solved the issue by removing the nesting. Here is the spec sample that generates issues.
spec_with_issues.txt

Sorry for the delay, @umnex. I've added a feature tag here for our Product team to review this as a feature request where the implementation will likely take time. It is a fixed limit of 10 levels of nesting I believe.

As mentioned in my last post, to get around these limits, I'd recommend checking out the openapi-to-postman project and this thread in particular where these limits are discussed for how to bypass them:
postmanlabs/openapi-to-postman#306 if that's a feasible workaround in the meantime?

Thanks for the update. I don't think I passed the 10 levels limit in my spec. Can you confirm it please?

@umnex sorry to be mistaken on my part! I spoke with @VShingala and the "Too many levels of nesting to fake this schema" problems are due to how big the spec is where there are only six layers of nesting.

As a workaround, you can turn off Optimize Conversion setting, which should prevent the deeper layers of nesting from being populated with "Too many levels of nesting to fake this schema". I verified this with your file as well.
Screen Shot 2022-05-30 at 2 32 13 PM

Going to close this out but feel free to reopen it if this issue is persisting! I'll log a ticket internally to try to better this error messaging where it's not intuitive to know to disable this Optimize conversion setting I realize.

Hi Team,

Turning off **Optimize Conversion** setting via UI does work.
Is there a way to do that via the Postman API?

I am using the Automation-API Workflow and still getting the error. Documentation is not clear/ doesn't show settings available in the UI

 URL

Turning off "Optimize conversion" worked for me... thanks a lot.

@umnex @timbochamp Just making sure this doesn't get lost in translation, this **works** if we were to use the POSTMAN UI to import, but **doesn't work** via API (no field to turn off optimize conversion. Any suggestions?

@pierre-wehbe have you tried openapi-to-postman as a workaround?
 URL

You should be able to set `optimizeConversion=false` that way I want to say!

Turning off Optimize Conversion doesn't work for me. I use collection from here: URL

I am also using the POSTMAN UI to import, and turning off the "Optimize Conversion" setting doesn't work. I am still getting the " " error.

I am also not able to tell if I've reached the 10 stack limit nesting as I am not sure how exactly that's measured. Are there any updates on when this feature will be implemented?

@Gunacan
I got a same problem and I found a solution!
It worked really well for me.

sh
openapi2postmanv2 -s root.yaml -o collection.json -p -O folderStrategy=Tags,includeAuthInfoInExample=false,optimizeConversion=false,stackLimit=50

@iksflow, thanks for your response, but as I said in my message, I am doing the import on Postman UI (Postman app), and not the API.

@Gunacan
Yeah, that's right. It's not cool.
You already know, So I imported the collection.json(generated by CLI command) to Postman App. 🤣

This ticket should not be marked closed as the root cause has not really been addressed.

Reopening to track with ticket internally to try to better this error messaging or arrive at a solution not requiring openapi-to-postman as a workaround

No, openapi-to-postman doesn't work for me. It's even worse because it generates only 2 levels when it comes to this situation.

Using Version 10.11.1 of Postman UI and the setting does not work for me.

Using Version 10.11.2 of Postman UI and the setting does not work for me.

I have come back to this issue after a period of time working on other things and have stumbled-upon a solution.

It seems that if you pass the config options on the command-line, e.g:
`openapi2postmanv2 -s postman.yml -o collection.json -p -O folderStrategy=Tags,requestParametersResolution=Example,optimizeConversion=false,stackLimit=50`
you get the " " error, but if you pass the same parameters in a JSON config file, e.g.
`openapi2postmanv2 -s postman.yml -o collection.json -c config.json`
where config.json contains:

{
 "optimizeConversion": true,
 "folderStrategy": "Tags",
 "requestParametersResolution": "Example",
 "optimizeConversion": false,
 "stackLimit": 50
}

it works as expected, with no error generated. In my case, it now claims to be using a stack of depth 11.

@maldaba-nigel I tried this out myself with stripe API which has maximum of 27 nested levels and using same args provided as you, it's generating collection without any `" "` error in collection.

Could it be the case that the OpenAPI definition you're working with has more than 50 levels of nesting?

@shae128 @pmrussell892 If possible you be able to share the definition / sample definition where this issue is still reproducible?

It's still possible to have this error if there is too much nesting in the data. i.e. more than 30 levels. But many cases are now generat8ing collection without this. For example, Here's Stripe API which used to have this limitation but with this change, generated collection from this now does not contain such error.

Is their any plans to allow users to change the stack limit using Postman settings?

@pmrussell892 We're looking more into this. Meanwhile, you can make use of our open-source collection generator from OpenAPI definition.

Here is more on how this tool can be installed and used. This conversion will take place on your machine so it'll be private. Do provide your feedback if increasing this limit is generating collection as expected or not.

Bumping this. It's been over a year since this last message:

How's the looking going?

@jstjoe With the current limit that we have in place (stack=30) most of the use-cases should not be having this issue. But if you have this issue occurring where even for a property that is not very nested and you receive this error, let us know and we can look into it.

As for exposing this option in App, It seems allowing this option could cause many potential issues due to sheer amount of collection size that few definitions that are imported can generate. We've seen collections getting generated that are more then 500 MB to even 1 GB in such cases. Due to this, we've made decision to allow such generation only from usage of module directly.

How do I get to this screen in current version of postman?

Image

As an example, in postman you see stuff like this

Image

This was imported from my openapi documentation json.
When examples or responses have deeply nested key value pais, you start seeing this error

Image

But in swagger, everything is displayed, and no errors

Image