Question: How to correctly taint "%this"?

How to correctly taint "%this"?
### Overall Description

Hi, I want to write a plugin that can taint any class when any field of the class is tainted (e.g, when a tainted variable is passed to `obj.setName(String)`, I want the `obj` is tainted as well after that). However, the caller variable is not affected even through the "%this" variable in the callee function is tainted. So how to correctly handle this?

You can refer to the minimum reproduce code below that the "%this" in callee function is tainted

However the caller "%this" is untouched

I may misunderstand some parts of the pta analysis, do I need to manually propagate this (why not be handled automatically)?

### Expected Behavior

The caller object is also tainted.

### Current Behavior

The caller object is not tainted.

### Tai-e Arguments

 Click here to see Tai-e Options

yaml
optionsFile: null
printHelp: false
classPath: []
appClassPath:
 - tester-1.0-SNAPSHOT.jar
mainClass: org.example.Main
inputClasses: []
javaVersion: 8
prependJVM: false
allowPhantom: true
worldBuilderClass: pascal.taie.frontend.soot.SootWorldBuilder
outputDir: output
preBuildIR: false
worldCacheMode: false
scope: ALL
nativeModel: true
planFile: null
analyses:
 ir-dumper: ;
 pta: "taint-config:config.yml;plugins:[pascal.taie.analysis.pta.plugin.taint.SuperTaintHandler];dump:true"
onlyGenPlan: false
keepResult:
 - $KEEP-ALL

### Tai-e Log

 Click here to see IR log

 static void process(java.lang.String r1) {
 org.example.User $r0;
 [0@L8] $r0 = new org.example.User;
 [1@L8] invokespecial $r0. ()>();
 [2@L9] invokevirtual $r0. (r1);
 [3@L10] invokestatic ($r0);
 [4@L11] return;
 }

 public void setName(java.lang.String name) {
 [0@L9] %this. = name;
 [1@L10] return;
 }

 Click here to see points-to results

[]: /%stringconst0 -> [[]:MergedObj{ }]
[]: /r0 -> [[]:EntryPointObj{alloc=MethodParam{ /0},type=java.lang.String[] in }]
[]: /$r0 -> [[]:NewObj{ [0@L8] new org.example.User}]
[]: /r1 -> [[]:MergedObj{ }, []:TaintObj{alloc= /0,type=java.lang.String}]
[]: /$r1 -> [[]:NewObj{ [1@L1147] new java.io.PrintStream}, []:NewObj{ [9@L1150] new java.io.PrintStream}]
[]: /$r2 -> [[]:MergedObj{ }, []:TaintObj{alloc= /0,type=java.lang.String}]
[]: /r0 -> [[]:NewObj{ [0@L8] new org.example.User}]
[]: /$r1 -> [[]:MergedObj{ }, []:TaintObj{alloc= /0,type=java.lang.String}]
[]: /%this -> [[]:NewObj{ [0@L8] new org.example.User}]
[]: ()>/%this -> [[]:NewObj{ [0@L8] new org.example.User}]
[]: /%this -> [[]:MergedObj{ }, []:NewObj{ [0@L8] new org.example.User}, []:TaintObj{alloc= /0,type=java.lang.String}]
[]: /name -> [[]:MergedObj{ }, []:TaintObj{alloc= /0,type=java.lang.String}]

### Additional Information

 Click here to see key code of my plugin

// It may not be efficient, any suggestions to improve?
@Override
 public void onNewCSMethod(CSMethod csMethod) {
 JMethod method = csMethod.getMethod();
 Context context = csMethod.getContext();
 IR ir = method.getIR();
 for (Stmt i : ir.getStmts()) {
 if (i instanceof StoreField) {
 StoreField st = (StoreField) i;
 if (st.getLValue() instanceof InstanceFieldAccess) {
 InstanceFieldAccess lv = (InstanceFieldAccess) st.getLValue();
 if (lv.getBase().toString() == "%this" && lv.getFieldRef().getDeclaringClass() == method.getDeclaringClass()
 && !st.getRValue().isConst()) {
 CSVar from = solver.getCSManager().getCSVar(context, st.getRValue());
 CSVar to = solver.getCSManager().getCSVar(context, lv.getBase());
 solver.addPFGEdge(from, to, FlowKind.LOCAL_ASSIGN);
 }
 }
 }
 }
 }

 Click here to see minimum reproduce code

package org.example;

public class Main {
 static void sk(User u){
 System.out.println(u.getName());
 }
 static void process(String src){
 User s=new User();
 s.setName(src);
 sk(s);
 }
 public static void main(String[] args) {
 process("xxx");
 }
}

package org.example;

public class User {
 public String getName() {
 return name;
 }

 public void setName(String name) {
 this.name = name;
 }

 private String name;

}

Hello,

Thank you for providing such a detailed description and information about the issue. This helps reduce the number of interactions needed, which is greatly appreciated by open source maintainers.

Before addressing your question, I have a side note. What prompted you to modify the placeholder for `Tai-e Log` in our _New Issue Template_? We intend it to provide runtime information originally, such as `Tai-e Commit: d610a880a2c05968c9e60400f2041f281dee809f` and `java.runtime.version: 17.0.6+10`, among other details. This is not a complaint, just a user study. 😆

---

My Intuition: In this points-to set, a _%this_ variable of type `org.example.User` points to a `String`, which doesn't follow the type system. You should mock another `TaintObj` with the same type as _%this_.

You mentioned "Any", so it is temporarily not achievable directly in the current Tai-e. Because _Writing taint configuration programmatically is our future plan. It's being incubated._

But if you are in urgent need, I provide a simple idea below for your customized implementation: monitor all changes in the points-to set of all InstanceFields. If a `TaintObj` appears, mock a TaintObj which pointed to by the InstanceField's Instance's var.

I found my `tai-e.log` is always empty :(. In case some information are needed:

commit: 47bdb8b2361083151a44ba76ee2f9f2dbd363b40
java: Java(TM) SE Runtime Environment (build 17.0.11+7-LTS-207)

 Thanks for your help. I wrote another transfer to do this:

public class ThisTransfer implements Transfer {
 HeapModel heap;

 public ThisTransfer(HeapModel heap) {
 this.heap = heap;
 }

 @Override
 public PointsToSet apply(PointerFlowEdge edge, PointsToSet input) {
 if (edge.target() instanceof CSVar && ((CSVar) edge.target()).getVar().toString() == "%this") {
 List append = new ArrayList<>();
 input.forEach(o -> {
 if (o.getObject() instanceof MockObj mo && mo.toString().startsWith("TaintObj")) {
 if (mo.getType() != edge.target().getType()) {
 append.add(new CSObj(heap.getMockObj(mo.getDescriptor(), mo.getAllocation(),
 edge.target().getType(), mo.getContainerMethod().orElse(null), mo.isFunctional()), o.getContext(), o.getIndex()));
 }
 }
 });
 append.forEach(input::addObject);
 }
 return input;
 }
}

and invoke in the plugin

solver.addPFGEdge(new PointerFlowEdge(FlowKind.LOCAL_ASSIGN, from, to), new ThisTransfer(solver.getHeapModel()));

[]: /name -> [[]:MergedObj{ }, []:TaintObj{alloc= /0,type=java.lang.String}]

From the log I can ensure the callee %this is tainted with correct type, however, the call site is

There is still no tainted object here. How can I notify or add the taint object to this? I think this should be handled by Tai-e automatically but something must be wrong.

What you do is `User.setName/%this /$r0 -> [NewObj]
 /%this -> [NewObj, TaintObj]

`$r0` and `%this` are two variables in different methods' IR. The `$r0` will propagate to `%this`; it is because `[2@L9] invokevirtual $r0. (r1);` create a PFG Edge from `Main.process/$r0` to `User.setName/%this`. However, `%this` will not backpropagate to `$r0`.

Can the configuration of source and sink support wildcard characters, such as using * to match？
### Clear and concise description of the problem

Can the configuration of source and sink support wildcard characters, such as using * to match

### Impact Analysis

_No response_

### Suggested Solution

_No response_

### Alternative

_No response_

### Intention to submit PR

No

### Additional Context

_No response_

Oh, it is a very valuable and funny issue.

As it happens, We're working on a more convenient and dedicated mechanism for describing class/method/field in taint configuration; the hard part is we're trying to balance readability and functionality when designing the "wildcard" expression mechanism. For example, regular expressions are powerful, but they are less readable; maybe we need support for describing subclasses, but more is not better, it depends.

Anyway, we will support it. Stay tuned for next release milestone.

Can we consider opening up an inheritable abstract class that can use Java to write rules, so that users can override and implement the logic in DeserializeSources, DeserializeSinks, DeserializeSanitizers, and DeserializeTransfers according to their needs?

Support for signature wildcards is now available. Documentation is currently in progress.

Here's a preview: URL

The taint rules with wildcard support are now released in v0.5.1. For details on using wildcards in signatures, please refer to our documentation on signature patterns.


Question: Docker container update status and update action?

@marcokreeft87 I created a docker insights module for the integration.

The following sensors gets created

1. Container CPU Usage
2. Container Memory Usage
3. Container Network Download / Upload
4. Total Containers
5. Running Containers
6. Paused Containers

Feel free to checkout the experimental feature and let me know what you think

You will need to copy the unraid folder from experiment/docker-insights branch to your home assistant instance.

You can only enable docker insights after the initial configuration setup and it takes about a minute to initialise and create the sensors.

Screenshot 2024-11-24 at 4 53 54 pm

You will see 2 device entries after initialisation

Screenshot 2024-11-24 at 4 55 05 pm

I'll look if its possible to get docker uptime data

Docker container update status and update action
**Is your feature request related to a problem? Please describe.**
I keep opening the Unraid interface mainly for the docker container updates. I have over 100 running at the moment so there is pretty much daily something to update, but I also want to be selective and not update something that I rely on at that moment in time. So I want to create some automations using Home Assistant.

**Describe the solution you'd like**
- A boolean flag in the attributes of a docker container if there is an update avaialble.
- An action that can be called on that container to update it.

**Additional context**
This would allow me to create automations for updating containers I know are not problematic automatically and show the ones that I need to time it right in some list on a daskboard for myself where I can hit a button and have them updated. If there is something I can do to help with this, please let me know (also a software dev here).

@ChristianRiesen Thanks for the feature request. I'm actually experimenting with a docker insights feature that's listed in URL

I could look at extending it to provide what you are after.

Feel free to try it out just remember to install the dockersocket proxy container on your unraid server first with network set to host. Also make sure to clear your browser cache if you see some sensors not showing correct data.

I tried to extend the docker socket proxy feature but that didn't work. Caused a lot of stability issues. So much that I'm removing docker insights and docker socket proxy.

Might be able to do it with docker commands but will increase the load on unraid and HA a lot.

You could use the docker engine API. Here is a SDK for it: URL
That way you can use it directly and much leaner then if you were to spawn commands over and over.

Feel free to fork the repo and try and implement this feature. If you can get it to work without causing HA instability issues then can merge it into main

Add docker containers stats
Can you please add information about the docker containers running in Unraid?
Like state, name ofcourse, CPU usage and such.

No rush :)

PS. My end goal is make a dashboard like this with your integration:

 URL

@marcokreeft87 I think I'll create a seperate docker monitoring module that pulls these metrics and have it as an option in the integration setup / configuration. Just in case no one else wants to see this type of metrics and extra sensors.

 - version
 - containers_running
 - containers_paused
 - containers_stopped
 - containers_cpu_percentage
 - containers_memory_percentage
 - cpu_percentage
 - health
 - uptime
 - memory
 - memory_percentage
 - network_speed_up
 - network_speed_down

Just installed it. But first issue I see is I have all new entities now for the old values:

image

And some are not working anymore:

image

I made updates to the code to fix a unique ID issue with sensors and it also now gets the unraid hostname to use with naming convention of sensors. This allows you to add multiple unraid servers without issues.

Unfortunately the update doesn't migrate existing entities to new ones.

Best is to remove the integration and start fresh. Sorry about that

I updated the sensors to look like this. 1 sensor per container with attributes.

Screenshot 2024-11-26 at 5 26 02 pm

I updated the code to use dockersocket container as it provides better results so you need to have dockersocket container running on unraid.
Screenshot 2024-11-26 at 5 29 51 pm

Im trying the latest version right now. The CPU temp was my fault btw :)

Do I need special settings for dockersocket?
image
Its not working currently

image

Oh stupid me. That coming from a guy that works with docker daily xD Thnx it's working now

I have resolved this now and docker insights feature will be available in next update. Pre-reqs will be to have dockersocket proxy container running on unraid.

I have also added fixes that it will load and unload faster without holding HA back from starting / restarting

I have decided to remove this Frankenstein of a feature as its causing too many stability issues and sleepless nights.

This integration achieves the same but has more options if someone wants have detailed container metrics.

 URL

But in the latest release the docker feature is working right?

Thanks for the 'frank' response and suggestion, I was having a nightmare with running out of sessions due to the number of queries being spawned (large unraid docker setup) thanks again. I think it is sensible to keep docker integration separate, if for only troubleshooting advantages.


Question: Time zone problem?

@buckaroo54 You're running a full sync or a delta sync? Either press `Shift` and click `Sync`, or use the dropdown on the Sync button...

Time zone problem
Depuis l'installation de la dernière mise à jour, les rendez-vous créés dans Google sont décalés d'une heure dans Outlook.
Par exemple, si je crée un rendez-vous à 10 h, il apparaît à 9 h dans Outlook. Par contre, la synchronisation dans l'autre sens fonctionne bien.

Since installing the latest update, appointments created in Google are shifted by one hour in Outlook.
For example, if I create an appointment at 10am, it will appear at 9am in Outlook. However, synchronisation in the other direction works fine.

Having the same problem here in the Netherlands (time zone Amsterdam; UTC +1). **Not only new Google meetings are shifted one hour back in Outlook, but also excisting ones!!**

Checked time zones in Outlook and Google, also the time zone mapping in OGCS. They all use UTC +1. No alterations made in the systems timezone settings.

Can't retrieve the date I upgraded to 2.11.2.0.

Pls, pls, pls advice! Happy OGCS user for years!

OGcalsync.log

Dear Paul @phw198 , thank you very much! It runs like clockwork again! 👍 #veryhappy

Thanks a lot…

Jean-Michel LÉVÊQUE

14 rue des Caves Basses

34210 CESSERAS

*06 45 52 06 47

***@***.*** ***@***.***>

De : Paul Woolcock ***@***.***>
Envoyé : jeudi 6 février 2025 19:14
À : phw198/OutlookGoogleCalendarSync ***@***.***>
Cc : Beth-Salem ***@***.***>; Author ***@***.***>
Objet : Re: [phw198/OutlookGoogleCalendarSync] Time zone problem (Issue #2065)

Duplicate of #2047 >

v2.11.2 falsely detecting start/end time changed (by number of hours offset from UTC)
**OGCS Version**: v2.11.2

**Bug Description**
OGCS incorrectly detects changes in start and end times for Google events, due to comparisons in UTC not localtime.

**Steps to Reproduce the Issue**
Full or delta syncs report a change in start and/or end time when there is none. The difference in time is the offset between localtime and UTC.

:package: This should now be resolved with hotfix v2.11.2.1.zip (see instructions for applying).
      CRCs: Zip = `F6A3BA15` Exe = `8197E679`

Please let me know how it goes, if you apply the hotfix.

I know I'm not the original reporter, but I too had this issue on 2 recurring meetings and 2 seperate meetings I'd added after moving to the new Aplha.
The fix has resolved the issue for me. The meetings have returned to their correct time.

I can also confirm hotfix resolved this issue. Needed to do a full sync to revert to the correct times.

I have an additional problem described here: URL
Even after the full sync, I still get these offset errors.
Also I think the time shift is sometimes still wrong. But have to check that again tomorrow. Will add more information, if it's really still wrong.

The hotfix corrected the bug for me. Thanks for all your hard work!

Thanks everyone for confirming the fix 👍🏻

As this is a high impact bug, I've quietly patched the ZIP and setup.exe on v2.11.2.0 for new installers; unfortunately upgraders will have to apply the hotfix on this page.

@lemonbutter we'll track your issue over on #1978, as it predates the bug that caused the problem on this issue.

Hi, this has fixed the problem when viewing items in the calendar that are synced from google cal to outlook cal.
The entries are now at the correct time - so amazing and Thankyou!

But there may still bepart of the bug not fixed - ie. in the csv's produced and in the OGCS sync verbose output, the timestamp on the text displayed/logged is still incorrect.
It seems to be not respecting the timezone offset - eg. in the outlook_create.csv, the "starttime" and "Endtime" shows eg. "5/02/2025 6:50:00 AM +00:00" which maybe in my example should really likely read as something like "5/02/2025 6:50:00 AM +08:00"?
And the sync log shows "And the OGCS app sync log tab (with verbose logging ticked) shows "5/02/2025 6:50:00 AM " but should show "5/02/2025 2:50:00 PM ".

@joedotbloggs You are indeed correct and I had already coded a fix, but just not built a hotfix for it as no one had spotted it yet!

:package: If you would like to confirm hotfix v2.11.2.6.zip does indeed resolve it, it would appreciated though (see instructions for applying).
      CRCs: Zip = `7BFFA637` Exe = `6A844883`

This hotfix (2.11.2.6) works - fixes both the CSV's and the Verbose Log!
Thank you :)

The hotfix is not fixing the issue for EXISTING items in Google. ONLY if I am changing them a bit - like adding 5 minutes and re-syncing - it is fixing the wrong entry in outlook!

@rafi-d For fixing already syncing items you either need to edit the Outlook item(s), or run a full sync from O->G.

What about from G to O? Your bug was in Outlook times (in my use case)

I'm still having the issue that @rafi-d had, with past appointments not updating. I tried hotfixes 2.11.2.1, 2.11.2.3, 2.11.2.4, and 2.11.2.6. It comparies all of the entries and says that none of them need to be updated, even though the past Google Calendar entries are in the correct time and the same past Outlook entries are not. The sync is set Outlook <-- Google. Any ideas?

I was _not_ running a full sync 🙃 Thank you for the guidance, all's well now! 😁👍


Question: Support working with query string and JSON body?

Thank you, could you give some example API requests and flows you have? I have a bit of a hard time following because I don’t know Centrifugo

Support working with query string and JSON body
### Preflight checklist

- [X] I could not find a solution in the existing issues, docs, nor discussions.
- [X] I agree to follow this project's Code of Conduct.
- [X] I have read and am following this repository's Contribution Guidelines.
- [ ] This issue affects my Ory Cloud project.
- [ ] I have joined the Ory Community Slack.
- [ ] I am signed up to the Ory Security Patch Newsletter.

### Describe your problem

Support working with query string and JSON body

### Describe your ideal solution

1. Mutating and hydrating query string and JSON body (e.g. adding/replacing/removing a query string parameter or a JSON body path)
2. Extracting data from query string and JSON body for keto_engine_acp_ory (e.g. data from a query string parameter or a JSON body path)
3. Support autherizer rules for query string parameters and JSON body paths (e.g. allowing or disallowing specific values)

### Workarounds or alternatives

-

### Version

-

### Additional Context

_No response_

Any updates on that? It seems like #843 is related to this.
We are more interesting in query params right now, will a PR will be accepted here?

I found a workaround for query params:

{{ .MatchContext.URL.Query.Get \"a\" }}

No code change is requires in oathkeeper to support it, just the power of Go templates 💪

Like this idea: @omerlh Did you find any issues with this at scale?

Have you had a chance to get the way how to extract some values from original request body?

If you use the `remote` authorizer the full request body is sent to this service so you can use the request body for authZ. We built a small service that handle authZ so it also handles this part.

Working with query parameters
** Is your feature request related to a problem? Please describe. **

We use the so called Centrifugo subscription server. It can authenticate socket or SSE subscription requests to a proxy such as this one. But it makes requests with a list of channels to subscribe only in parameters, not in path.

** Describe the solution you'd like **

So that the outkeeper accepts channel parameters from the request and can use them in the request to the authorizer.
For example, a GET request came / path / parameters (chanel1, chanel2). The outkeeper mutates such a request (if required) and you can or cannot contact the authorizer using the parameter values to obtain approval.

** Describe alternatives you've considered **

Define channels in the form of paths on the outkeeper, mutate them if the request passed the check, make the request further through the hydrator.

api example.

We have next config structure:
match:
 url: URL

authorizer:
 handler: remote_json
 payload: {channels: "{{ .RegexpCaptureGroups 1 }}"}

And for request:
 URL channel2, channel3]

Authorizer gets object: {channels: "channel1, channel2, channel3"}

Would the workaround described here solve/close this issue?
 URL


Question: Add automated test for Epinio UI?

First steps are done, one test was triggered last night and all was green!
 URL
Let's create other small cards to avoid this one to become too big.
Fixed by URL

Add automated test for Epinio UI
We have to find a way to do automated tests on the Epinio UI.

There is multiple ways to do this:
- only the API (like other Rancher products)
- as a real user (with something like Selenium)
- both

Let's discuss the pros and cons of all and anyone can add his ideas here also ;-)

Good progress last week, we are close to the end of the first version I think (improvements will be needed):
Cypress_ok
We are now more familiar Cypress so coding new scenarios is faster.

I think we can close this epic, upgrade and CI will be done in there own cards.

Epinio UI test - Cypress CI integration
We have to configure CI for our Cypress tests:

1. What components versions we have to test?
In a first time: main branch everywhere, last stable rancher-dashboard

2. When do we want to run tests?
Every night: URL

3. Which repo can trigger tests?
epinio-server? helm-chart? e2e?
Let's start with epinio/epinio, it's the faster way to start (self runners already there etc...)
**EDIT:** Some runners are not shared across the Epinio organization, so let's try in epinio-end-to-end-tests repo.

4. Which web browsers to test?
Firefox, Chrome, Safari (mac runner?), headless mode

5. One workflow by scenario

6. Linter to check Cypress code?

 URL
Rancher + Epinio in the same node for using only one runner (self hosted), Cypress runs through Docker images (on self hosted runners too).

Today, I've pushed a first test and all is green!
 URL
Now I have to add more scenarios and improve workflows.


Question: Symmetries of a C2v system inconsistent with PySCF 2.0 and NWChem?

Using the testing script from #1464 , it's not hard to see that the px orbitals of the oxygen atom is assigned to B1 symmetry, and py orbitals are assigned to B2 symmetry. I believe the irreps are correctly labelled. I'll mark this issue as solved for the moment. Feel free to reopen the issue to continue the discussion.

Symmetries of a C2v system inconsistent with PySCF 2.0 and NWChem
See issue #1464. . The snippet from that issue, on the current master branch, returns:

['A1' 'A1' 'B2' 'A1' 'B1' 'A1' 'B2']
['A1' 'A1' 'B2' 'A1' 'B1' 'A1' 'B2']

The output for the unshifted and shifted molecule is the same, but it is the "incorrect" output, i.e. what Harry got for the "unshifted" molecule and not consistent with NWChem (or PySCF 2.0).

PR URL changes the behaviour to be inconsistent with previous versions of PySCF and with NWChem.

@sunqm can you comment? Is the new behavior preferred, even though it is inconsistent with previous versions and with NWChem?

The symmetry detection code in PySCF 2.0 and older versions incorrectly assigns the axes for C2v molecules. It was fixed in #1201. I don't know which convention NWChem follows. I'm sure the symmetry irreps in PySCF 2.1 and newer versions are consistent with the C2v character tables from various sources, such as URL , URL . See also the discussion in the issue #1430 .

I'll mark this issue as solved for the moment. Feel free to reopen the issue to continue the discussion

Character table of C2v group seems incorrect
In here, it seems B1 and B2 need to be swapped.

From Housecroft 'Introduction to Molecular Symmetry'
image

A quick look at other sources confirms that you are right and the table in pyscf is wrong.

sx in the code means the plane perpendicular to x axis. So it is consistent with your table. I've added some comments in param.py .

@fishjojo , thanks for raising this issue. Has it been resolved?


Question: [Bug]: Current integ tests workflow fails on version incompatibility?

The cmake upgrade to 3.23.3 version was initially setup as a follow up in the al2023 upgrade/nodejs upgrade, tho now it seems like a 2.9.0 dependencies thus it needs to be updated soon.

Thanks.

[Bug]: Current integ tests workflow fails on version incompatibility
### Describe the bug

The current OS integ tests fails on importing module because `urllib` only supports `OpenSSL` 1.1.1+. Current ssl version is `1.0.2`

Link: URL
Full context:

Running ./src/run_integ_test.py manifests/2.8.0/opensearch-2.8.0-test.yml --component job-scheduler --test-run-id 4980 --paths opensearch=/var/jenkins/workspace/integ-test/tar --base-path URL ...

Traceback (most recent call last):

 File "./src/run_integ_test.py", line 13, in

 from test_workflow.integ_test.integ_test_runners import IntegTestRunners

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/integ_test_runners.py", line 11, in

 from test_workflow.integ_test.integ_test_runner_opensearch import IntegTestRunnerOpenSearch

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/integ_test_runner_opensearch.py", line 15, in

 from test_workflow.integ_test.integ_test_suite_opensearch import IntegTestSuiteOpenSearch

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/integ_test_suite_opensearch.py", line 22, in

 from test_workflow.integ_test.topology import NodeEndpoint, Topology

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/topology.py", line 16, in

 from test_workflow.integ_test.local_test_cluster import LocalTestCluster

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/local_test_cluster.py", line 13, in

 from test_workflow.integ_test.service import Service

 File "/var/jenkins/workspace/integ-test/src/test_workflow/integ_test/service.py", line 14, in

 import requests

 File "/usr/share/opensearch/.local/share/virtualenvs/integ-test-B5RGX0jA/lib/python3.7/site-packages/requests/__init__.py", line 43, in

 import urllib3

 File "/usr/share/opensearch/.local/share/virtualenvs/integ-test-B5RGX0jA/lib/python3.7/site-packages/urllib3/__init__.py", line 39, in

 "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently "

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips 26 Jan 2017. See: URL

script returned exit code 1

### To reproduce

 URL

### Expected behavior

_No response_

### Screenshots

If applicable, add screenshots to help explain your problem.

### Host / Environment

_No response_

### Additional context

_No response_

### Relevant log output

_No response_

This seems like an upgrade on urllib3 libs due to not supporting openssl version lower than 1.1.1+, where centos7 does not have now. We would need to speed up the process to upgrade to rockylinux8.

The newer version of requests cause a requirement update on openssl, from 1.1.0 to 1.1.1+, this means CentOS7 is not able to run newer version of request on any activities related to our python code.

Hi,

Option 1: Revert #3544 and keep using CentOS7 for this release
Option 2: Move all the changes to RockyLinux8 and remove CentOS7 support.

Consider our decision on the AL2023/NodeJS18/PythonUpgrade 3.9 has already made in this comment: URL

We will go ahead with option 1 and revert #3544.

cc: @bbarani @gaiksaya

Thanks.

Our test in #3544 is running in ubuntu-latest thus does not have the openssl issue in centos7, therefore, not caught.

[Proposal] Support AL2023/NodeJS18 for OpenSearch/Dashboards Releases
With the announcement of Amazon Linux 2023 Preview available for testing, we want to setup some notes here for the support of AL2023 once it is fully released to public.
 URL
 URL

* Evaluate the compatibility of AL2023 when running OpenSearch/Dashboards
 * OpenSearch/Dashboards latest binary can run on AL2023 with the same expected behaviors on AL2
 * Working alongside plugin teams to ensure all the plugins are running as expected
 * Performance testing to ensure there is no performance regression
 * Evaluate the package/dependencies availability on AL2023 default repository
 * Evaluate the size of the image vs contents integrity
* Evaluate AL2023 availability
 * AL2023 available on docker for both x64 and arm64
 * AL2023 available on EC2 for both x64 and arm64 for testing
* Evaluate Build Process
 * What level of changes do we need to support AL2023 with current build process and scripts
 * Can we support both AL2 and AL2023 in the same system
 * Build tool related libs/packages/dependencies availability on AL2023
* Evaluate deprecation plan for older OS
 * When would we deprecate AL2 while supporting AL2023
 * Will this introduce breaking changes to customer
 * Would we support older version of OpenSearch/Dashboards with new OS?

We welcome more discussion on this.

Thanks.

Hi @nknize @dblock what do you think about this change?

From our teams perspective it makes sense to upgrade the main docker image from AL2 to AL2023 starting 3.0.0. It also make sense that we use this opportunities to upgrade other OS such as Ubuntu and Rockylinux to their latest versions.

Thanks.

I really don't know enough about this @peterzhuamazon. What are pros/cons?

Now with this issue on the horizon, we need to also think about how to proceed with the changes:
* opensearch-project/opensearch-dashboards#3601

This means:
* We have to deprecate ci image that uses centos7 or al2.
* Our ci and release docker images have to move from al2 to al2023
* #3351
 * Python upgrade from 3.7 to 3.9 must follow as al2023 is default to newer version of python
 * AL2 has python 3.7 by default but AL2023 is already bundled with 3.9. It would take longer time just try to compile Python 3.7 on AL2023.
* KNN will 100% break for certain old OS users if we still use the same docker image for 1.x / 2.x / 3.x, as centos7 is the pinnacle of KNN, as it has the oldest glibc version which does not support Node18.
 * URL
 * Unless we add a new jenkinsfile just for 1.x, we cannot tweak the existing dist-build-dashboards jenkinsfile too much as it is near the size limit of Jenkinsfile max size.
 * Adding a new jenkinsfile just for 1.x will break a lot of the internal hardcoded folder structure on S3
 * One way to avoid this is that OS has to use centos7 to build and OSD has to use al2023 to build, which is very unformalized and hard to maintain.
* This has to happen after Jenkins Upgrade complete
 * URL
* This is a major breaking change and seems like they are trying to backport to 2.x as well due to node14 is deprecated and we cannot ship a deprecated product.

For the Node 18 upgrade, we're targeting 2.8. This is important as Node 14 no longer will receive security updates (as of April 2023). Is there an ask to delay to solve the above?

KNN support on the native code compilation:

node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by node)
node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by node)
node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by node)
node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by node)

# CentOS8/RockyLinux8
ldd (GNU libc) 2.28

# CentOS7
ldd (GNU libc) 2.17

# AL2
ldd (GNU libc) 2.26

# AL2023
ldd (GNU libc) 2.34

Seems like if we want to switch from CentOS7 to CentOS8, we will need to remove support of CentOS7 and AL2 from our support matrix on our website as well, since CentOS8/RockyLinux8 is the baseline of node18 with glibc 2.28.
 URL

 URL
cc: @krisfreedain

I would agree @peterzhuamazon - if we test on CentOS8/RockyLinux8 the compatibility matrix needs to be updated to accurately reflect that for the community

Hi @peterzhuamazon we will bump the node.js to version `18.16.0`

@peterzhuamazon want to clarify, you are saying that node 18 requires 2.28, correct? Then doesn't this break compatibility with AL2 and centos 7? So, regardless of k-NN libs, upgrading to Node 18 would require us to remove centos 7 and AL2 from the compatibility matrix.

If this is not the case, we could also elect to compile k-NN libs in separate docker image to maintain compatibility.

@peterzhuamazon not sure if you saw this URL

unlike chromium, it is possible to do a custom build of nodejs on centos 7 to lower glibc dependencies (I also saw musl builds somewhere, which can be another possibility to workaround this issue). I've not looked into performance or other issues might be caused by downgrading glibc, but if OS upgrade is difficult maybe this would be easier?

Did anyone confirm node 18 with AL2? @seanneumann

We noticed some glibc related compatibility issues. Not sure if it's an isolated incident.

That is likely not an option because CentOS7 is going to be out of support in 1 year, and we are also on track to move to al2023 anyway. Manually rebuild nodejs18 would add another dep on our side, which is not going to scale on the long run.

Thanks.

Node 18 will not run on AL2; I tried it.

OSD 2.8 will be compatible with Node 14, 16, and 18. OSD 2.8 will also bundle Node 18 and a Node 14 for fallback; with this OSD 2.8 can run on AL2 by using the bundled Node 14.

1. It does, al2/centos7 have older libs than 2.28.
2. Agreed that we need to move out of centos7 and al2 at some point.

If node14 out of support can we even bundle a 14 alongside 18 together?

@peterzhuamazon I hate it but being a minor release, we have to.

Need to move the integTest runner to rockylinux8 as well:
* URL

These three will need to get baseOS change to rockylinux8 before adding node18.
These are the only ones do not have node18 on docker images, AMI not affected.

docker/ci/dockerfiles/current/build.centos7.opensearch-dashboards.x64.arm64.dockerfile
docker/ci/dockerfiles/current/release.centos.clients.x64.arm64.dockerfile
docker/ci/dockerfiles/current/test.centos7.performance-test.x64.arm64.dockerfile

Rockylinux:8

gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Will switch to devtooset8 on centos7 to match with rockylinux:8 which we will eventually switch to.

Using scl and devtoolset might be a problem because it requires running `source scl_source enable devtoolset8` either through rc or through profile.

With us locking docker images to non-interactive shell this is not the best option.
Exploring manually compile gcc 8 from source.

Hi All,

We are going to stay with node16 for 2.9, and plan to move to node18 for 2.10.
This would ensure us to have enough time for the upgrade.
As well as compensate the changes for multiple teams.

We can also start the Python3.9 upgrade process as Python3.7 is about to be eol by 2023/06/23.

Once we go to node18 in 2.10, the support matrix of our product will need to remove these OSes:

* RHEL7&CentOS7 / AmazonLinux2 (glibc2.17 / glibc2.26)
* Ubuntu 16.04 / Ubuntu 18.04 (glibc2.23 / glibc2.27)
(Note: we still bundle node14 as fallback so technically above can still run, but 14 is eol and we do not officially support them anymore)

This also requires that the 2.x branch stays on 16.20.0 in .nvmrc, and have no node18 specific code committed until 2.9.0 is released to the public.

Infra will take care of the node18 upgrade once after 2.9.0.

Thanks.

cc: @CEHENKLE @dblock @seanneumann @AMoo-Miki @ananzh @seraphjiang @vamshin @jmazanec15 @bbarani @hdhalter

## 20230801 Update: (AL2023/NodeJS18 on 2.10.0 Release)

### Required Steps

* Update all AMIs to have NodeJS14/16/18 support.
* Update all CI docker images to have NodeJS14/16/18 support.
* Update all Release docker images to use NodeJS18 as the base node version for OSD.
* Deprecate all CentOS7 images in build systems and replace with the RockyLinux8 images (Except client release image as it does not require nodejs18 by default for now).
* Work with OSD team members to ensure all OSD core and plugins are locking to NodeJS 18.16.0 at least on 2.x/2.10 branches. 1.x can pending.
* Since KNN already update the dependencies such as gcc to v7 and cmake to 3.23.3, it is easier for us to bring them onto either Rockylinux8 or AL2023. Starting from now k-NN will not be able to run on OS with glibc version lower than 2.28 (previously 2.17)
* Work with doc team to update the doc compatibility charts to remove CentOS7 support and add AL2023.
* Test all the pipelines on the new servers.

### Optional Steps
* Need to discuss after the 2.10.0 release whether or not 1.x line need to upgrade as well.
* Have separate pipeline to test NodeJS14 bc abilities.

What is the status of the separate pipeline to test node 14 backward compatibility? There are going to be users using old linux versions who can't go past node 14 yet, and it would be useful to know what fixes we can do so as to not unnecessarily break things for those users.

@wbeckler We do not have plans to create a separate pipeline in public Jenkins infra for EOL versions of Node. If needed, this needs to be tackled at repo level.

Have quick talk with OSD @AMoo-Miki about core, and we are planning to switch 2.x to 18.16.0 next week to start testing on the 2.10.0 build with newer nodejs.

Thanks.

The majority of the tasks in this issue is resolved and implemented.
We will issue new tickets whenever new tasks coming up.

Thanks.


Question: Automatically creates a replication of nodes in the scene when the game is closed?

At a glance I suspect the issue is that the change in timing means the scene editor adds a preview environment or similar incorrectly and then doesn't restore it, could be related to incorrectly not checking ownership as well

Automatically creates a replication of nodes in the scene when the game is closed
### Tested versions

the master version

### System information

windows 10

### Issue description

 URL

### Steps to reproduce

Follow the actions in the video

### Minimal reproduction project (MRP)

test1.zip

Your video doesn't work, and please specify what version you are on specifically

Also sounds like:
* URL

From just the description, without the video

I don't know why the video won't play, it's just a simple mp4 video, but it can be played after downloading

Then this problem is solved in the latest version, try building the latest version

If you're still having problems please comment and we can investigate if further

Switching scene tabs duplicates nodes in the scene and eventually also leads to an editor crash.
### Tested versions

- Reproducible: 4.3dev master[99ff024f78f65ba0bc54fb409cfeca43ba2008fe]
- Not reproducible: 4.3 dev5

### System information

Windows 11 - Vulkan - Nvidia RTX 4070 - intel i5 13600KF

### Issue description

When switching scene tabs while running the project some nodes in the scene tree get duplicated and stay in, eventually leading to a crash after couple times of repeating.

**Video:** URL

### Steps to reproduce

1. Open the MRP.
2. Open the Test.tscn to have two tabs (main and test).
3. Go to main and press save and reload the project (this step is just in case, might not be necessary).
4. Play the project and keep the play window somewhere (doesn't matter).
5. In the editor switch from main to test tab and then back to the main tab.
6. Observe some nodes got duplicated.
(7. Repeat this proces to duplicate more - you might need to always press the stop and play buttons for every duplication - and observe after couple duplications the editor hard crashes)

### Minimal reproduction project (MRP)

Mrp.zip

EDIT(@lyuma + @Illauriel): simpler MRP and repro steps:
Another MRP: node-dupe-bug.zip
Steps to reproduce:
1. Open node3d.tscn.
2. Open the nested scene.
3. Save the nested scene
4. Return to the original node3d.tscn. it will contain duplicate nodes.

The reason play duplicates nodes is because the play button saves all scenes. Therefore saving is sufficient.

Regression from:
* URL

Unsure if this *caused* it or just exposed an existing issue that didn't fire without this

CC @ajreckof @KoBeWi

You don't need to play the project. Just save the instance scene and go back to the parent scene. This causes reloading the scene (to update the instance) and leads to the mentioned bug.

After enough repeats, I got the editor to freeze while spamming `Object was deleted while awaiting a callback.` in the console.

In my own project (not the MRP) it duplicates several nodes as seen here (check the Time label is the only one without @ in the name but it was also duplicated).
image

Well the crash seems specific to your scene. The bug is that nodes get duplicated, but depending on nodes, the end result might be different.

It's due to an incorrect deletion sequence in the PR, writing a fix right now:
 URL

Indeed, this should go from last to first.

But this approach feels hacky in the first place, maybe we could add a bool to `replace_by` to not copy child nodes?

@Illauriel and I confirmed a bisect of #89447
Another MRP: node-dupe-bug.zip

Steps to reproduce:
1. Open node3d.tscn.
2. Open the nested scene.
3. Save the nested scene
4. Return to the original node3d.tscn. it will contain duplicate nodes.

I also had a suspicion that an internal node somewhere could cause an issue: I recall from debugging a different issue that the core function `Node::replace_by` may be buggy in the presence of internal nodes (off-by-n error), since it does not properly check indexing, so seeing `replace_by` in the regression PR is setting off that alarm bell as well.

From my notes about replace_by:

All nodes have to be added and moved to the same index, one by one. This might make files with thousands of nodes slow.
Calling replace_by on an internal node will malfunction because it does not preserve the internal_node in the add_child call
I do think there's a core bug in that replace_by is incorrect for internal nodes

I should try and repro + file the bug about replace_by, since it's probably a separate issue, but wanted to bring it up anyway just in case.

Moving a `GDScript` from one folder to another also reproduced this bug for me.

It seems that it will appear as long as you save the sub-scene and switch to the main scene.

Nodes will be duplicated only when the nodes meet the following conditions:
1. Is a direct non-internal child node of the main scene root;
2. The node index is an even number.


Question: Python 3.13 not support?

Issue gh-88279 was in 2021.

I wrote it in this issue description in 2023:

What changed in that the PyConfig API is now widely available and is a better API than the deprecated one.

Python 3.13 not support
Hi there!

Something got updated on my system, and I am unable to build master, as well as branches that I have been able to build before.

In all cases, the error message is this:

[ 10%] Building CXX object CMakeFiles/vegastrike_python.dir/src/python/config/python_utils.cpp.o
/mnt/int/system/aur/vegastrike-engine-git/src/Vega-Strike-Engine-Source/engine/src/python/config/python_utils.cpp: In function ‘const std::string GetPythonPath()’:
/mnt/int/system/aur/vegastrike-engine-git/src/Vega-Strike-Engine-Source/engine/src/python/config/python_utils.cpp:44:37: warning: ‘wchar_t* Py_GetPath()’ is deprecated [-Wdeprecated-declarations]
 44 | wchar_t* w_path_ptr = Py_GetPath();
 | ~~~~~~~~~~^~
In file included from /usr/include/python3.13/Python.h:123,
 from /usr/include/boost/python/detail/wrap_python.hpp:178,
 from /usr/include/boost/python/detail/prefix.hpp:13,
 from /usr/include/boost/python/args.hpp:8,
 from /usr/include/boost/python.hpp:11,
 from /mnt/int/system/aur/vegastrike-engine-git/src/Vega-Strike-Engine-Source/engine/src/python/config/python_utils.cpp:31:
/usr/include/python3.13/pylifecycle.h:46:43: note: declared here
 46 | Py_DEPRECATED(3.13) PyAPI_FUNC(wchar_t *) Py_GetPath(void);
 | ^~~~~~~~~~
/mnt/int/system/aur/vegastrike-engine-git/src/Vega-Strike-Engine-Source/engine/src/python/config/python_utils.cpp: In function ‘PyObject* GetClassFromPython(std::string, std::string, std::string, std::string)’:
/mnt/int/system/aur/vegastrike-engine-git/src/Vega-Strike-Engine-Source/engine/src/python/config/python_utils.cpp:76:5: error: ‘Py_SetPath’ was not declared in this scope; did you mean ‘Py_GetPath’?
 76 | Py_SetPath(python_path);
 | ^~~~~~~~~~
 | Py_GetPath
make[2]: *** [CMakeFiles/vegastrike_python.dir/build.make:79: CMakeFiles/vegastrike_python.dir/src/python/config/python_utils.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:224: CMakeFiles/vegastrike_python.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
==> ERROR: A failure occurred in build().
 Aborting...

Normally I have makeflags -j32, so this failure got jumbled among all the other jobs that are running, but for this build single jobs were specified.

With a little Googling, it seems related to this commit:

 URL

Hi there...
I could change the way that vegastrike builds on Arch linux, and link to an older version of python, probably. However, this is an issue that needs fixing sooner or later, as other distributions of Linux will be following suit and also updating their version of Python to 3.13.

Keeping this in mind, is there any point in making a special package that uses an older python?

I guess it boils down to how long and involved this issue is to fix. I'm not a good enough coder to know, does anyone else have an idea?

@evertvorster agreed; though it may be a week or so before we get to it. You might be able to temporarily.

Okay, so I just looked at the code. This bug appears to be fairly limited in scope, with no more than 4 or 5 lines of code calling any of the deprecated or removed Python APIs from that deprecation. All of these lines of code were introduced in two of your commits, @royfalk , one from 2024-10-02 and the other from 2024-11-01. Filenames:

1. init.cpp
2. python_tests.cpp
3. python_utils.cpp
4. python_utils.h

The calls in question are Py_GetPath and Py_SetPath. No other deprecated APIs are involved here.

It looks like these two APIs can be replaced using URL

Please note, also, that this problem does not exist on the 0.9.x branch, since the Py_GetPath and Py_SetPath calls hadn't been introduced yet.

Not sure what changed, but git master compiles again.
There are a LOT of warnings, but it does compile.

Sweet! Yeah, I thought I had fixed it, sometime in the last week I believe. That's good to get confirmation that it is fixed.

C API: Remove legacy API functions to configure Python initialization
The following C API functions were deprecated in Python 3.11 by issue #88279:

* ``PySys_AddWarnOptionUnicode()``
* ``PySys_AddWarnOption()``
* ``PySys_AddXOption()``
* ``PySys_HasWarnOptions()``
* ``PySys_SetArgvEx()``
* ``PySys_SetArgv()``
* ``PySys_SetPath()``
* ``Py_SetPath()``
* ``Py_SetProgramName()``
* ``Py_SetPythonHome()``
* ``Py_SetStandardStreamEncoding()``
* ``_Py_SetProgramFullPath()``

IMO it's now time to remove them: the new PyConfig API is available since Python 3.8: URL

### Linked PRs
* gh-105154
* gh-105179

Oh no, too bad, I forgot to deprecate PySys_ResetWarnOptions() in Python 3.11.


Question: design: is the buffer of a string or a bytes value required to terminate with a 0 ?

I agree with what Michael says (and think it's consistent with `writef()/readf()` where ultimately we want to do more compile-time checking for param formatting strings, but having any checking is better than none).

I don't think there should be a back door for users, but that having a "prevalidated" code path for internal use makes sense.

I don't have a strong feeling about validating within the I/O system vs. at string creation time at present. Whatever we do, I'd like to not have to support three different string validation pieces of code. Ideally, they'd all use the same code (which might suggest doing the validation in C so that it can be reused in the compiler and via the runtime?).

design: is the buffer of a string or a bytes value required to terminate with a 0 ?
[Update:] This started as a bug issue assuming that the the answer was "yes". Turns out that we are not settled on the answer.

Therefore this issue now requests to determine the answer and to adjust our implementation to comply with it. If the answer is "yes", users can create non-compliant strings (see below). If the answer is "no", we likely have code that will malfunction when passed a non-0-terminating string.

[Previous contents:] Our convention today for the underlying buffer of a string, i.e., `_string.buff` (source), is that it is zero-terminated. Therefore, s.buffSize should be at least s.buffLen+1.

Neither condition is checked in `createStringWithBorrowedBuffer(x: c_ptr(?t), length: int, size: int)` (source). Therefore users can invoke this function in a way that creates a string with our convention violated.

The test `test/library/standard/IO/readAll/readBytesArray.chpl` (source)does exactly that. It invokes createStringWithBorrowedBuffer() passing it a buffer of size 1032 and length == size == 1032. The violation of both conditions goes unnoticed because nothing in that test relies on the invariant to hold. Whether or not the string is 0-terminated is immaterial because the buffer `a_signed` does not have room for that final zero.

If we switch the test to invoke createBytesWithBorrowedBuffer(), it will have the same violation.

Chapel version: 4d5379da31

We have the choice to introduce the checks in createStringWithBorrowedBuffer / createBytesWithBorrowedBuffer vs. `initWithBorrowedBuffer(ref x: ?t, other: bufferType, length:int, size:int)` (source) . I suggest the latter if it is user-facing. Note that the former (the string version) already validates for UTF8-ness.

We also have the choice to disable this validation upon --fast / --no-checks. I can go either way, given that it is constant-time. Although checking for the terminating zero may be a cache miss.

While there... we may want to optimize getHash() to return 0 if x.buffLen == 0 and countNumCodepoints() to return buffLen if it is <= 1, to avoid potentially going cross-locales in these cases.

It's my understanding that we are trying to null-terminate all of our strings in memory, but I don't think it's necessarily the case that we should guarantee that property in a user-facing way.

In the long term, the main issue with null-terminating all strings in memory is that it prevents us from having a string slice that refers to another string's memory. (See also #11222 and #12670 about how we might one day add "borrowed strings" which would make this relevant). But, even in that case, if we are willing to add a bit to the string record, we could arrange for the string to manage allocating a copy of itself that is null terminated (and deleting it when the string is deleted). Relatedly, today we use `someString.localize().c_str()`; here the temporary copy of the string (if needed) is created because `someString.localize()` returns a new string by value. Perhaps we could generalize this pattern to effectively do `someString.localizeAndZeroTerminate().c_str()`.

I updated the issue to determine whether we require null-termination and to adjust our implementation to comply with the answer.

Still catching up on December, I'm tagging @e-kayrakli as string-/bytes-master to get his read on this.

I don't have strong opinions on the matter.

First, I certainly hope that none of our methods on string/bytes rely on null-termination. I know that the way we have nulls at the end is a bit haphazard.

Some cases where enforcing null-termination can be problematic:
- As Michael pointed out string views can be a little difficult to implement when/if we need them,
- Related but, cases where we want to borrow a part of a raw buffer as a Chapel string/bytes
 - e.g., here we don't get the null-terminator from qio interface, but it is fine. Should we allocate a new buffer and add null at the end? But that's not borrowing. So should we not be able to create "borrowed" strings: URL
 - e.g., do we need to avoid null-termination for exporting string/bytes arguments? I can't explain what's going on here, but it is another case where a non-null-terminated "borrowed" string is created: URL

I don't think these cases are without solutions should we enforce null-termination. But the design around creating "borrowed" strings from raw data would have to change.

However, having null-termination guarantee can make C interop easier. I am curious whether there's any internal case where we have to add/ensure null termination before calling an extern C function.

Arguably, `.c_str()` does rely on the string being null terminated.

True. I meant more from an algorithmic standpoint. We shouldn't be relying on a terminating null to determine the end of a string. It should all be based on the length.

Are you suggesting that it is OK for a Chapel string to lack the terminating 0? I am game for that. In this case, how to implement .c_str() ?

I was merely saying that the implementation of string algorithms do not rely on the null termination, but instead they work with the a priori knowledge of string's length.

Thinking about this more, one of the main points that I was thinking about is that `0x00` is legal UTF-8 byte ( URL Granted, it is only used for the null codepoint and no other codepoints. So, another way to ask the question this issue is asking is "can Chapel strings contain `0x00` byte?"

Currently, the answer to that question is "yes". This is consistent with python, which was the main example while we were converting our strings to unicode-only. However, this is not good for C interop, at least not today. `"AAA\x00AAA"` will not be usable with C interop:

chapel
extern proc printf(s...);
var s = "AAA\x00AAA";

writeln(s); // AAAAAA (my terminal doesn't show null bytes)
printf("%s\n", s.c_str()); // AAA

Options as far as I can see:

1. Make an exception for `0x00` and not support it within Chapel strings
 **Con:** This might hurt python interop
 **Pro:** Better C interop

2. We can make all our string null-terminate *and* keep a field as to whether the buffer contains `0x00`, in which case `c_str()` halts
 **Con:** Potentially costly, especially for `bytes` for which there is no validation today.
 **Pro:** Can interact with C and Python well

3. Make `c_str` return a copy of the buffer with `0x00` filtered out except the terminating one
 **Con:** `c_str` will be costly
 **Pro:** Not breaking
 3a. Same as 3, but do filtering only if a field like in (2) says the string contains null byte
 **Con:** Costly for `bytes`, like (2)
 **Pro:** Not breaking, not bad for `c_str` performance

4. Keep the status quo
 **Con:** buggy C interop
 **Pro:** ?

Of those I'd go with the status quo. I think C programmers are used to the limitation that C strings cannot contain null bytes (other than the null terminator). They can use a pointer + length strategy when the code needs to handle nulls. We can support calling `extern procs` with a pointer + a length for our C strings (e.g. `c_ptrTo(myString)` can return a pointer to the buffer; and we have a function to get the length in bytes already, I think).

I *am* suggesting that Chapel strings should not be required to always be null-terminated. My best ideas for adding the null for `.c_str()` are:
 1. `someString.localizeAndZeroTerminate().c_str()` or
 2. the proposal in #21233

Does it, or does it do a deep copy of the string? (at which point it could add the null). I was wondering about this while trying to figure out how Scott's NetCDF code today, and couldn't easily come to a conclusion by just reading the code, though it seemed like it might be a deep copy (unless `const` intent means `const in`, which I doubt).

It does not. It returns the pointer to the beginner of the data buffer. It only casts it to `c_string`. (Where do you see `const`?)

The `const` was in Scott's blog article—I couldn't figure out how his code could work if something wasn't making a deep copy, either `.c_str()` or `const`.

closing as we decided to **not require** null termination on string/bytes buffers

UTF-8 validation for strings
Chapel strings are UTF-8 encoded. However, currently we don't check whether a given byte buffer is a valid UTF-8 sequence while creating a string.

The general consensus is that we should add this check during string creation. Interestingly, doing so can allow us to cache some properties of the string (number of codepoints, whether it is all ASCII etc) to improve performance for some common operations.

Exactly when should we validate?
--------------------------------

There are two main ways a string is generated in Chapel. String literals, and
I/O. They both end up calling an appropriate string factory function at the end.
So there are two alternatives on implementing UTF-8 validation.

1. String creation time: This is in the common path for both ways. So, it means
less implementation and maintanance effort. Another benefit is that if we want
to piggyback some metadata caching about the string (whether it is ASCII, or
some information about multibyte codepoints).

2. Separately in I/O and parser: This has the advantage of giving an error early if
necessary. Arguably, long string literals are unlikely so maybe it doesn't make
much difference for parsing string literals. But it may matter if you are
reading a long file and there are invalid bytes right in the beginning. Another
advantage may be giving a compile time error if the source file contains strings
literals with non-UTF-8 bytes.

Going through some of the past discussions, it seems that the agreement was that
we are going with the first option.

What is the behavior for when there are invalid bytes in a string?
------------------------------------------------------------------

If we are going with option 1, then we'll need to throw an error. Currently we
have a `DecodeError` which is thrown when you attempt to decode a `bytes` with
invalid byte sequence to `string`. Should we use the same error type? Or do we
see UTF-8 validation as a different (more intrinsic to the language may be?)
error and should create a different error type.

One other alternative is to rename it so that it explains the issue more clearly
for either case. `StringDecodeError` can be an option and I don't think it'll
create any confusion if it comes from `bytes.decode`

Should we add a "back door"?
----------------------------

We may consider allowing the users to opt out from validation. Most obviously
this can allow them to get rid of the validation overhead if they know the input
string to be UTF-8 in the first place. But can we support something like that
for string literals? If so how? We can consider adding an environment variable
to control the overall behavior. This is also related to #12728 and to some
extent adjusting the Chapel environment to use UTF-8 encoding.

Note about NULL bytes
------------------------
NULL byte is valid ASCII and therefore UTF-8 and Chapel strings and bytes
currently can contain NULL bytes. Strings will continue to do so even after
validation is implemented. C (and maybe other) interoperability layers should
avoid using `strlen`, `strcmp` etc functions that rely on NULL-terminated
strings.

Caveats
-------

- `string.localize()` creates a copy of the string. If the string was local to
 begin with, this is done through calling `createStringWithBorrowedBuffer`
 which re-validates the string. We may want to do something better than
 `localize` in the future. For now I see three alternatives:

 1. If we do not have a back door, then, a string is validated already.
 Therefore, we don't call validate if `createStringWithBorrowedBuffer` is
 called with a `string`

 2. If we add a back door, then, we should keep a `var valid: bool` in string
 record.

 3. Do nothing and accept that we may validate a buffer that has already been
 validated. (I am not a big fan of this option)

- It looks like the current QIO decode logic we have may not support some UTF-8
 sequences. This may need to be adjusted. One example that I bumped into is the
 replacement character `\uFFFD`.

#12723 suggests a test which can probably be part of this effort

#14325 Makes some changes to the `bytes` implementation that are necessary for this to move forward

Re where to validate:

It seems obvious to me that somebody creating a string literal containing invalid UTF-8 would want the error to occur at compile-time. Having program start-up fail and instead throw an error due to invalid UTF-8 in a literal seems unfortunate.

Here I think that neither of the options presented are sufficient. I think that the compiler should arrange to call an internal/private function for the string literals which does not do validation. Other than that, I think that we should start by doing the validation in the string validation functions. Since not all string creating happens through I/O, this covers more cases than 2. describes.

In the future, we might have the I/O system take on the task of ensuring that any read data is valid UTF-8. Arguably it already tries to do that (but we would need to do more testing and make adjustments - the issue above already noted an issue with the UTF-8 replacement character). If that happens, I would expect the I/O system also to use an internal/private function to create the strings in a way that does not do validation in order to avoid redundancy. However, the user-facing string factory functions would still do validation in that event.

Summing it up - I think validating in the string factory functions is a great starting point, and from there we should validate literals/param strings at compile-time and then validate read string data in the I/O code (and then arrange to avoid redundant validation if there is any benefit to doing so).

Just to be clear, the following is not a back door (simply because it is not documented) right?

chapel
var x: string;
x.buff = my_c_arr_that_contains_non_utf8_sequences;

Arguably you can break many things if you play around with the internals of types and UTF8 validation will only be an example of that?

I think more or less this is something that we can use internally to avoid redundant validation. For example, when we are returning from `bytes.decode()` we don't need to validate the string because we expect the decoded buffer to be valid utf8.

Right, I'm saying that I don't think the language should support an officially sanctioned back door (at least, I haven't been convinced of that yet). If we have one for our own development purposes, don't document it, and a user chooses to step into it, I don't think that's a problem (for us... it may be a problem for them if we change things down the road). In this case, I assume it's relying on the existence of a `.buff` member within string that is undocumented and therefore arguably being abused (?).


Question: Does this component support encryption and decryption for big data, Hive, hbase and other components?

# Change Scope
I check the documents and the source code, and I may miss something, feel free to let me know.
## Parser
### Add a new module shardingsphere-sql-parser-hbase
- antlr4: don't need to write all rules, can add more rules if needed
- hbase parser package:
 - `HBaseLexer`
 - `HBaseParser`
 - `HBaseParserFacade`
- hbase visitor package:
 - `HBaseDMLStatementSQLVisitor`: only support DML in phase1
 - `HBaseStatementSQLVisitorFacade`
- resources: HBase SPI config

### Add a new package in shardingsphere-sql-parser-statement
Parse statements and segments

## Add a new DataType
- Implement `DatabaseType`
- Implement `DialectTableMetaDataLoader`

## Proxy
- Implement `DatabaseProtocolFrontendEngine`
- Implement `JDBCDriverURLRecognizer`

# Questions
I don't know if we can use some frameworks like Apache Phoenix to help us convert the SQL to HBase queries. But if we choose to use Apache Phoenix, it looks like shardingsphere only need to do a few things. Or shardingsphere has to "compiles it into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets"

Does this component support encryption and decryption for big data, Hive, hbase and other components?
## Question

**For English only**, other languages will not accept.

Before asking a question, make sure you have:

- Googled your question.
- Searched open and closed GitHub issues.
- Read documentation: ShardingSphere Doc.

Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot reproduce it on current information, we will **close it**.

- HBase should be allowed to use only SQL92, but such unit tests have never existed, nor have antlr4 files that deal with Hbase's proprietary syntax.

- I think this issue is blocked by URL . If you're happy to submit some PRs to push for better unit tests, it may be resolved sooner.

I do not understand the principle of ShardingSphere. May I ask whether using ShardingSphere can achieve seamless interception of agents directly from the bottom layer, or do I need to change the service configuration or code to access the agent layer?

- For a general RDBMS, ShardingSphere Proxy will virtualize a data source, and whether it uses the MySQL dialect or the PostgreSQL dialect depends on your choice. ShardingSphere JDBC does the same, but the dialect used depends on the connected database.

- For HBase, I have the impression that the master branch never existed unit tests. For Apache Phoenix unit tests, everything with NoSQL is a bit unknown.

This issue has been inactive for a long time, so I will close it.
Please pay attention to #14061 for future updates.

shardingSphere For HBase
## Feature Request

It is hoped that proxy is the only SQL entry. Businesses only need to access proxy and deal with various databases later. For example, HBase is supported. In the process of use, there is no need to use HBase protocol to operate the back-end database
### Describe the feature you would like.
Support SQL statements

SQL HBase
show /*+ hbase */ databases -> cluster
show /*+ hbase */ tables -> list
show /*+ hbase */ create table t_test -> describe "t_test"
select /*+ hbase */ * from t_test -> scan "t_test"
select /*+ hbase */ count(*) from t_test -> count "t_test"
update /*+ hbase */ t_test set age = 18 where rowKey = 1 -> put "t_test"
delete /*+ hbase */ from t_test -> deleteall "t_test"
insert /*+ hbase */ into t_test (rowKey, v1, v2) values(1, 2, 3) -> put "t_test",1,'rowKey:v1', 3
...

Great!
It is very exciting to explore OLAP integrate with ShardingSphere.

I'd like to join building this feature. But it's a big feature, can you guys help to split it into several tasks? So I can help to do these tasks step by step.

Some initial tasks:

1. SQL parser of HBase SQL dialect
2. Recognize HBase JDBC url and driver
3. Parse hint by `/+ hbase */` from SQL
4. Route hinted SQL to HBase cluster

We can complete the more tasks here.

ok, I will try the first task, but it's a hard job for me, so it will take some time.

Yes, we can use Apache Phoenix to convert the SQL if needed.
Could you provide the design if we using Apache Phoenix in ShardingSphere?


Question: Cannot add liquidity with asymmetric ratio?

closing as asymmetric adds are currently disabled. Flat fees is introduced in order to re-enable asymmetric adds. The issue is not expected to appear once asymmetric adds are re-enabled

Cannot add liquidity with asymmetric ratio
Give a description of the bug! Include things like how to reproduce it, what you expected to happen, and what actually happened.

Cannot add to pool ROWAN/UFO. "There was an unknown error." Error failed to execute message; message index: 0: Cannot add liquidity with asymmetric ratio.

hey, can you try again? it's a known issue, and it's due to the fact that we disabled asymmetric adds

this is the same issue as:
395 and 389.

What's our strategy here? @bksifchain (My understanding is that we won't re-enable asymmetric liquidity adds anytime soon)

@hungng157 i have flagged this issue on the **eng-support** slack channel

Unable to add Liquidity Addition on Cosmostation
**Description:**
Unable to add liquidity symmetrically in Cosmostation for the SHIB pool, as an error log is triggered as the liquidity addition is asymmetric
Screenshot_20220610-015152

**Ways to reproduce it:**
Adding liquidity symmetrically on the SHIB - ROWAN pool

the enabling of asymmetric liquidity adds is on pause at the moment. To tackle this issue, it is suggested to try multiple times

to check if Formula has been updated in order to tackle the issue for Low Liquidity Pools


Question: [New game]: ShootOut?

Hey @kunjgit ! Thank you so much for your raising the issue💗
 It’s all yours, you can come anytime again and make some contributions! 🚀
 Alone, we can do little, but together we can do so much! 😇

[New game]: ShootOut
### 🎮 Game Request

This is a one-player shooting game made with HTML, CSS, and JavaScript. You will get four difficulty levels, and with each increasing difficulty, the speed and number of enemies will increase. With the help of three different weapons, you can kill enemies and score points. According to the number of enemies killed and weapons used score will be awarded.

I am further looking to improve this game but as of now I want to contribute this to GameZone

### Point down the features

- Made with HTML, CSS, and JavaScript.
- Animations and Sound Effects
- Four different difficulty levels
- Three different weapons
- Score Board

### Select program in which you are contributing

GSSoC23

### Code of Conduct

- [X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @apurva-1403 !
 Thank you for raising an issue 💗
 You can self assign the issue by commenting /assign in comment 😀
 Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
 Don’t Forget to ⭐ our GameZone🎮
 Make sure you join our Discord🕹

Hey @apurva-1403 👋!

 @kunjgit here,
 Hope you are doing extremely well 😇
 It's confirmed 🥳, GameZone is taking part in hacktoberfest and ready to have another 💥
 Ready to have some amazing contributions once again also thank you for you support in GSSoC23 you have took us on top 5 ❤🔥
 if you still haven't joined us join us on Discord
 See you soon 😊

Hey @apurva-1403 👋!

 @kunjgit here,
 Hope you are enjoying this hacktober 😀
 We are excited to share that we are opening issue from our side where you will be going to add games on our website 🚀
 Work on this issue 🔥
 and help us building GameZone 💗
 if you still haven't joined us join us on Discord
 See you soon 😊

Hello @apurva-1403, Time's Uppp!⏰
 Sorry for closing your issue!
 But it's more than a week since we haven't received anything from your side 😢 .
 Come up with new ideas, create a new issue and make sure you finish it within a week! 🔥
 All the best! 🚀
 Happy Hacking! 💗

Hey @apurva-1403 ! Thank you so much for your raising the issue💗
 It’s all yours, you can come anytime again and make some contributions! 🚀
 Alone, we can do little, but together we can do so much! 😇

[Enhancement]: Adding Games on Website 🔥
### **You have to add all the games which is there on GameZone and not added in website :smile:**
Step to complete this task :blush:

1. Finding all the data of games that are not added in website till the time you are adding it (you have to refer included_games and also the README )
2. Getting the data of games through the pr they made to add that game (tip just refer to that game you will find the pr )
3. Extracting the data for each game you want to add and add it in the GameData

"1": {
 "gameTitle": "Master Typing",
 "gameUrl": "Master_Typing",
 "thumbnailUrl": "Master_Typing.webp"
 },

- make sure you are adding the image and gameurl properly there

4. update the counter value in card_generator
5. **run everything locally and check website for the same and make sure that everything is working perfectly** :fire:

And.......................... you are good to goo :partying_face:
make a PR for the same and we will review and merge it sooonnnnn :rocket:

Hey @kunjgit !
 Thank you for raising an issue 💗
 You can self assign the issue by commenting /assign in comment 😀
 Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
 Don’t Forget to ⭐ our GameZone🎮
 Make sure you join our Discord🕹

Hey @kunjgit 👋!

 @kunjgit here,
 Hope you are enjoying this hacktober 😀
 We are excited to share that we are opening issue from our side where you will be going to add games on our website 🚀
 Work on this issue 🔥
 and help us building GameZone 💗
 if you still haven't joined us join us on Discord
 See you soon 😊


Question: Community Standards?

Yes, please!

They come up now when you open a new issue or pull request. You can give it a try and just click through it without adding anything.

I'm all for it! Unfortunately, we have ~1500 Doxygen warning in `svFSI+` so we can't do that anytime soon. But we could try implementing something that the number of warnings must go down in every pull request.

We could do the same for coverage. StanfordSpezi uses Codecov, which @MatteoSalvador and I briefly looked into. However, it's only free for public repositories (`svFSI+` is still private).

Community Standards
I just stumbled upon the Community Standards page of this repository. There are some good ideas in there (which we already want to implement @MatteoSalvador, @alisonmarsden), with examples:
- Code of conduct
- Contributing (in addition to the examples we have from openCARP and OpenFoam)
- Issue templates
- Pull request template

We can start adding those to this repository and then adapt them for "larger" projects (like `svFSI+/-`).

Yes agreed - we can develop these here and then migrate to other projects.

Closing this. Community standards will automatically be adopted once this repository moves to URL (see URL

Add developer guidelines to the website
We need to a page to the website covering guidelines for developers and contributors to SimVascular. This should include:
1) code of conduct
2) contribution guidelines

I added the following Community Standards:
- Code of conduct
- Contributing guidelines
- Issue templates
- Pull request template
- Introduction to SimVascular GitHub
- License

Please have a look, @alisonmarsden, @MatteoSalvador, @menon-karthik, @ktbolt. These are meant to be a first draft so we have something to start with. The files can be easily edited in the browser.

The files are all in a separate .github repository within SimVascular. This way, all repositories default to those standards (but can overwrite them with their own).

All files are heavily inspired by BioDesign's StanfordSpezi software project (with community files here). **Many thanks to @PSchmiedmayer for the inspiration and GitHub guidance!** 😄

It will be great to have these up on our website before the CZ software grant is due on Oct 17th. This is one of the items they ask about.

From: Martin R. Pfaller ***@***.***>
Date: Friday, October 6, 2023 at 3:47 PM
To: SimVascular/simvascular.github.io ***@***.***>
Cc: Alison Lesley Marsden ***@***.***>, Mention ***@***.***>
Subject: Re: [SimVascular/simvascular.github.io] Add developer guidelines to the website (Issue #8)

I added the following Community Standards:

 * Code of conduct

I will rewrite the `Contributing Guidelines` , get rid of all this

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119]

and remove all of the words using all capital letters like `MUST` below

You MUST follow our Code of Conduct.

writing in all capital letters is like shouting

Thanks @mrp089! I went through all the community standards. I just have a few suggestions for the moment:
- **Issue templates:** it might be useful to give the possibility to the user to attach several types of files (pictures, scripts, meshes, `XML` configurations) in order to ease the bug fixing process.
- **Pull request templates:** I would clearly state that adding tests for each new feature implemented is mandatory. I would also mention the importance of code coverage during testing, which should ideally always increase after any PR.
- **License:** I have always reported this type of MIT license at the beginning of each `.h`/`.cpp` file. I am particularly referring to the sentence

I am of course happy to keep discussing this!

This looks like a really great first draft @mrp089, thanks for doing this!

I noticed some typos in the `Contributing Guidelines`. Should the rest of us fork/edit/merge the usual way or does this repository work differently?

About the issue and pull request templates, this is probably just me being dumb, but it's not clear to me how this actually works. When a new pull request/issue is created does it automatically open up in the specified template format for the user or does a user have to do something to get that format? If the latter, maybe we should provide instructions.

I like @MatteoSalvador's idea about adding specific instructions for tests and code coverage, but I also think we should actually enforce it by using Doxygen to prevent merging undocumented code. And mention that we are doing this in the instructions. See discussion here. What does everyone else think?

Thank you very much for the constructive feedback, @MatteoSalvador and @menon-karthik! I'll add my replies below. I'm closing this issue so we can continue the discussion in URL

@alisonmarsden, I've added a brief `Contributing` section to the SimVascular homepage and linked the Contributing Guidelines and the Code of Conduct (#10).

Great idea, please add!

Also great idea, please add!

Happy to discuss in URL


Question: Backend tests with the "force-dynamic" mode are not executed by "cmake check-onnx-backend-dynamic"?

I am closing this issue, since ll issues errors with dynamic backend tests (on CPU) are fixed. Thanks!

Backend tests with the "force-dynamic" mode are not executed by "cmake check-onnx-backend-dynamic"
(Updated in March 6th, 2024)
Currently "make check-onnx-backend-dynamic" does not test dynamic cases but tests static cases, because "test/backend/common.py" sets the "TEST_IMPORTER_FORCE_DYNAMIC" environment variable, but "test/backend/variables.py" does not process it.

PR#2735 fixes this issue, and dynamic backend tests are enabled.
It also disabled the following 27 backend tests to pass the Jenkins backend tests temporally.

Your supports to fix these issues are very welcome!
If you fix some issues or find some unnecessary tests, please report the results at this Issue. Thanks.

1. test_instancenorm_epsilon_cpu
2. test_instancenorm_example_cpu
3. test_gathernd_example_float32_cpu
4. test_gathernd_example_int32_batch_dim1_cpu
5. test_gathernd_example_int32_cpu
6. test_castlike_DOUBLE_to_FLOAT_cpu
7. test_castlike_FLOAT_to_DOUBLE_cpu
8. test_convtranspose_3d_cpu
9. test_convtranspose_cpu
10. test_constant_pad_cpu
11. test_convtranspose_pad_cpu
12. test_convtranspose_kernel_shape_cpu
13. test_convtranspose_output_shape_cpu
14. test_einsum_transpose_cpu
15. test_einsum_inner_prod_cpu
16. test_depthtospace_example_cpu
17. test_convtranspose_1d_cpu
18. test_convtranspose_dilations_cpu
19. test_convtranspose_autopad_same_cpu
20. test_convtranspose_pads_cpu
21. test_einsum_batch_diagonal_cpu
22. test_depthtospace_crd_mode_example_cpu
23. test_einsum_batch_matmul_cpu
24. test_einsum_sum_cpu
25. test_edge_pad_cpu
26. ~~test_onnxmlir_top_k_float16_cpu~~ # Test condition fixed. K should be a1D tensor, cannot be dynamic.
27. ~~test_onnxmlir_top_k_smallest_float16_cpu~~ # Test condition fixed. K should be a1D tensor, cannot be dynamic
28. test_reflect_pad_cpu
29. test_spacetodepth_example_cpu
30. ~~test_top_k_smallest_cpu~~ # Test condition fixed. K should be a1D tensor, cannot be dynamic
31. ~~test_top_k_cpu~~ # Test condition fixed. K should be a1D tensor, cannot be dynamic
32. ~~test_top_k_negative_axis_cpu~~ # Test condition fixed. K should be a1D tensor, cannot be dynamic

@cjvolzka GroupnormalizeOp was just an error example. Other tests also failed before the PR#2725, too.

Closing this issue, because it is replaced by a new Issue #2743.

Dynamic backend tests cause errors.
(Updated in 2024/05/08)
All issues errors with dynamic backend tests (on CPU) are fixed. Thanks!

(Updated in 2024/03/26)

PR #2735 enables dynamic backend tests, but the following backend tests still fail.
Accordingly PR #2735 temporally disables these failed dynamic backend tests to pass the Jenkins tests.

Please look at the failed tests if you have time. We may have to (1) fix the issues, (2) change the test conditions, or (3) remove the tests if we can confirm that they are not supported.

Please check the following marks if fixed some of them. Thanks.
PR #2753 executes the failed dynamic backend tests, and you can look at the error log at URL .

[X] 1. "test_castlike_FLOAT_to_DOUBLE_cpu" # Fixed by PR#2762
[X] 2. "test_castlike_DOUBLE_to_FLOAT_cpu" # Fixed by PR#2762
[X] 3. "test_castlike_FLOAT_to_FLOAT16_cpu" # Fixed by PR#2762
[X] 4. "test_castlike_FLOAT16_to_FLOAT_cpu" # Fixed by PR#2762
[X] 5. "test_castlike_FLOAT16_to_DOUBLE_cpu" # Fixed by PR#2762
[X] 6. "test_castlike_DOUBLE_to_FLOAT16_cpu" # Fixed by PR#2762
[X] 7. "test_convtranspose_1d_cpu"
[X] 8. "test_convtranspose_3d_cpu"
[X] 9. "test_convtranspose_autopad_same_cpu"
[X] 10. "test_convtranspose_cpu"
[X] 11. "test_convtranspose_dilations_cpu"
[X] 12. "test_convtranspose_kernel_shape_cpu"
[X] 13. "test_convtranspose_output_shape_cpu"
[X] 14. "test_convtranspose_pad_cpu"
[X] 15. "test_convtranspose_pads_cpu"
[X] 16. "test_depthtospace_example_cpu"
[X] 17. "test_depthtospace_crd_mode_example_cpu"
[X] 18. "test_einsum_batch_diagonal_cpu"
[X] 19. "test_einsum_batch_matmul_cpu"
[X] 20. "test_einsum_inner_prod_cpu"
[X] 21. "test_einsum_sum_cpu"
[X] 22. "test_einsum_transpose_cpu"
[X] 23. "test_gathernd_example_int32_cpu" # Fixed by PR#2748
[X] 24. "test_gathernd_example_float32_cpu" # Fixed by PR#2748
[X] 25. "test_gathernd_example_int32_batch_dim1_cpu" # Fixed by PR#2748
[X] 26. "test_instancenorm_example_cpu" # Fixed by PR#2763
[X] 27. "test_instancenorm_epsilon_cpu" # Fixed by PR#2763
[X] 28. "test_constant_pad_cpu" # Fixed by PR#2754
[X] 29. "test_edge_pad_cpu" # Fixed by PR#2754
[X] 30. "test_reflect_pad_cpu" # FIxed by PR#2754
[X] 31. "test_spacetodepth_example_cpu"
[X] 32. "test_top_k_cpu" # Fixed by PR#2735
[X] 33. "test_top_k_smallest_cpu" # Fixed by PR#2735
[X] 34. "test_top_k_negative_axis_cpu" # Fixed by PR#2735
[X] 35. "test_onnxmlir_top_k_float16_cpu" # Fixed by PR#2735
[X] 36. "test_onnxmlir_top_k_smallest_float16_cpu" # Fixed by PR#2735

(This issue replaces the issue #2639 , since the situation is changed by PR #2735.)

Update on the 2 instance norm failing tests. When I compile a model with dynamic shape, and then set the shapes at runtime, it works. But when we compile a model with dynamic shape AND set the size of these shapes by a compiler option, the compiler exit with an error. @chentong319 has a similar problem with another program, and he will test his solution (setting the output type to the same as the original type). Once Tong confirms that this work on his case, will do the same on these 2 cases of instance norm.

@chentong319 thanks for the suggested fix, it worked. PR #2763 implements the change for instance norm cases (added also group norm which needs the same pattern, even if it may not be tested at this time).


Question: Bottom standoffs - Bug or user?

The predefined case pull down is not meant to hold all possible cases. With thousands of possible cases that is not practical or possible with the limitations of the Customizer UI within OpenSCAD. The pull down menu of predefined cases are custom cases or ones that have additional changes from the accessories subsystem. In addition, a reference shell case is included as a benefit for all SBC from the OEMs that support this project by supplying SBC samples, Hard Kernel and Pine64.

This tool is meant to autonomously build cases for any supported SBC by selecting the SBC and base case design under the View tab. It will automatically create the smallest possible symmetrical case using the current default tab settings. Any additional changes desired can also be made from the respective tab default settings e.g. the type of standoff fasteners to use for the top and bottom.

This is explained in the README.md file and I would strongly recommend you pay special attention to the notes-tips-and-things-to-check section for other suggestions of things to consider when setting up a case. Any case you create can then be saved in the predefined case pull down menu by pressing the + sign next to the 'save preset' button near the top of the GUI. The + sign creates a new entry and the 'save preset' button saves the current selected preset. There is also an area in the README.md that covers what each GUI tab and its parameters accomplish.

So if you want an OrangePi shell case select the correct SBC under the View tab and then select the case design. It will be automatically created and any other changes you want can be selected from the appropriate tab for the default presets. It's that simple. FYI, What you are calling the default RPI case is not the default case. It is an example of a custom RPI5 case incorporating a M2 HAT accessory.

If you need further assistance reopen this issue or create another.

Bottom standoffs - Bug or user
OS: OpenSUSE Leap 15.6
OpenSCAD: 2021.01
SBC_Case_Builder 3.0 - current github.

First let me thank you for this great tool! I very much enjoy working with it, although I'm new to it an just started with my first project.

I designed an Extended Case for a Radxa 4c+ v1.41 using the base case design „shell“.
Here’s my workflow:
 1) Created a new model for the 4c+ based upon rpi4b in SBC_Model_Framework/sbc_models.cfg.
 2) „Integrated“ it following the instructions using get_customized_values.scad
 3) Verified the model by 3D printing a regular shell case.

But the bottom standoff support is cutting out material from the wall which results in a thinner wall up to the height of the full standoff. The workaround is changing the bottom standoff size support from 10,00 to 9,00. Cause the position of the standoff is defined by the pcb it’s not moving anywhere => problem fixed.

In the case of extended case bottom standoff this doesn’t work. Changing the ex case bottom standoff size, say from 10,00 to 8,00 moves the whole standoff closer to the corner still cutting material from the wall.

This is not happening with the top standoffs for both sbc and extended. Changing the model i.e. to rpi4b or c1+ shows the same behavior.

Is this a bug or is it me? It looks like a boolean operation should be union but is cut. Can I fix this in one of the OpenSCAD files? If yes, where.

I also would like to contribute the Radxa 4c+ v1.41 model. What’s the process?

Thanks for opening an issue. Looking at the code there is a bug in the Extended Case Standoff placement. As you already figured out, the PCB Standoffs are fixed to the PCB hole locations and work as designed. Changing the PCB Standoff Support Size is the way to address the issue you encountered.

The extended case standoffs were originally fixed as well, at 10mm from the corners. When variable standoff sizes(3mm, 2.5mm, etc) were implemented their location was changed to be relative to the Standoff Support Size. This change allowed them to be placed closer to the corner of the case and why they move when you change the support size. I 'll make any corrections that are needed for the Extended Standoff placement and post the fix once completed and tested. It's late evening here now so I anticipate 24 hours time should be sufficient unless I run into some unforeseen problem.

Regarding the Union verse Subtraction, a hole is first created in the case that is slightly smaller then the Standoff Support Size and then the standoff geometry is placed at the same location. This is required so that the type of fastener end( blind, countersunk, recessed, etc) can be accommodated in the case geometry, otherwise they would be covered by the case geometry and not visible.

Regarding contributing your Rock 4c+ model, that would be welcomed and appreciated. Knowing that Radxa doesn't do a very good job with their documentation IMO, I am confused. There is already a Rock 4c+ in SBC Model Framework which I thought was version 1.4. So are these corrections to the existing model, a new model version or something else? If I remember correctly the current model was created based on the STEP model from their Wiki and is unverified(yellow) and may need corrections. Regardless, it would be good to have a verified model of the Radxa 4c+ since I don't own one, so your help in adding or updating the current version would be appreciated. The program sbc_models_viewer.scad within SBC Model Framework can be used to view the current models using a customizer GUI.

Regarding the process, SBC models for SBC Case Builder are added to SBC Model Framework in a separate repository. The technical process to add a model you seemed to have figured out already, add appropriate entries to sbc_models.cfg. For reference, there is information in the README.md for SBC Model Framework that explicitly explains the technical process.

So to add or update a model, start by forking SBC Model Framework, add or modify the SBC model in sbc_models.cfg and then issue a Pull Request. I'll handle the changes necessary for SBC Case Builder and update along with the sub-module linkage to pull the latest version of SBC Model Framework. Once this is done any one that installs either will get the updated models. Those with existing SBCCB installations will need to update SBC Case Builder and it's sub-module SBC Model Framework, as noted in the SBC Case Builder README.md. If it is easier for you, instead of a PR you can open an issue in SBC Model Framework and post your entry from sbc_model.cfg for the Rock 4c+. The only down side with this approach is you wouldn't be shown as a contributor to the project.

Ok, I changed the code so as not to interfere with case side wall geometry for any of the standoffs. Don't forget to save your case entry in the sbc_case_builder.json file, if you saved it, otherwise it will be over written when you update SBC Case Builder.

Also let me know if you have any more problems and the answers to my questions above about the Rock4c+ model. If your print is good I'll change the status to verified(green) based on the answer to the model questions. If you haven't found the current Rock 4c+ model in SBC Case Builder take a look at my response to this past closed issue.

Many thanks for the incredible fast reply and fix! Just saw it on my mobile. I will test tomorrow ;-) In regard to adding the 4c+ v1.4 model I will give it a try as you described. I will also check the .step file they provided. The major difference though was a few mm here and there plus the stacked USB ports being on the wrong side. Stay tuned ;-) Thanks again!

Case selection very limited
I have just installed the lastest version of OpenSCAD for Windows (tried by 32- and 64-bit) and cloned the SBC project, did the init and the update. However, I have a VERY limited selected of SBCs. There's probably about 25-30 or so showing in the drop-down. There are no RPi (other than the default) and no OrangePi, for example.


Question: [AR] ASSISTENZA: allegati?

Riapro la issue perchè non vedo risolto il `BUG` sul caricamento dei png, o ora anche dei PDF. Ho provato sia con screenshot nominati corretti che con immagini scaricate native png senza problemi nel nome e:
- o blocca il caricamento stesso
- l'antivirus dice file corrotto

Allego i file caricati come allegati
image

[AR] ASSISTENZA: allegati
### Il caricamento degli allegati nn rispecchia il componente previsto del design system (variante Trascina e rilascia)
che mostra il tipo di file e il peso (elementi da mostrare per accessibilità), e quindi porta a vari problemi:

- testo placeholder dentro il campo input > Seleziona il file
- una volta caricato il file il campo Descrizione è uguale al nome del file e se molto lungo porta la tabella a scrollare in orizzontale e nascondere la CTA di eliminazione
- [x] Se carico più di 3 file dovrebbe essere **nascosto**/disabilitato l'input per selezionare file successivi invece che lasciar selezionare il 4 file e poi mandare in errore

La richiesta è di implementare il corretto componente da design system

image

Sarebbe coerente mantenere lo stesso approccio ogni qual volta ci siano allegati da caricare, vedi #240 e #229

@sguerra issue completata, non posso cambiare lo stato "In stage" con questo utente
ncapozzi

Chiudo la issue ma la questione sul design system rimane aperta.

[AR] CAMBIO RESIDENZA: tab Allegati
Nel momento in cui provo a caricare un allegato PNG derivato da uno screenshot, il caricamento non funziona.
PNG scaricati da internet invece funzionano.

 URL

Questo uno dei file derivati da screenshot e che ora va in errore proprio.
anscval-int anpr interno it_serviziCittadino_privacy (29)

Buongiorno @steps88,
il problema è dovuto alla presenza del carattere "." nel nome del file che può stare solo prima dell'estensione ".png". Pensiamao di aggiungere una frase nella form del tipo "Nel nome del file non sono ammessi i caratteri punto ..."

va bene, meglio ancora se si può aggiungere un controllo del nome una volta caricato il file, e nel messaggio d'errore

20241021-design-system-italia
serviziCittadino_servizi_richiesta_cambioresidenza_130165
certificato_22014428.pdf

@steps88 attualmente non ci risultano problemi nel caricamento allegati e verifica antivirus sia per pdf che png. Il giorno della segnalazione c'era un problema sull'antivirus ed in più uno dei file .png che cercavi di caricare non era veramente .png, ma .avif. (quello con l'immagine di design system Italia) Comunque puoi riprovare e farci sapere