Question: Feature request: Code generate support R language (RCurl package or httr pachage)?

@nviraj Thank you for sharing this. This was helpful.
We have decided we'll add support for both. Because even ~370k downloads a month is a good number.

We'll keep this thread up-to-date with the progress and also share sample snippets.

Feature request: Code generate support R language (RCurl package or httr pachage)
1. Postman Version: Postman v4.10.7
 2. App (Chrome app or Mac app): Windows APP
 3. OS details: Windows 7 / 10 64 Bit

could it possible to add R Language support in code generation function?
For example, could support generate code for R Language's RCurl or httr package.

We'll be soon adding this support. You can track the progress here: URL

Support for Rcurl and httr are now in the Postman platform

<img width="482" alt="image" src=" URL

Add R to Generate Code Snippets
Hi, Please can you add an R code snippet in the Generate Code Snippets section. Thank you for your help.

The specific R libraries this should be implemented for are httr and curl.

+1

The auto-generated R SDK by OpenAPI Generator seems pretty heavy so I would prefer code snippets using curl instead

Would really love this to be honest!
I encounter a lot of Postman collections in the wild but translating them manually to httr/ curl in R is a bit painful.
This would be a really beneficial addition to the Postman ecosystem considering R is at rank 12 according to TIOBE index.

Thanks for feedback and this feature request. I am happy to let you know that we will be soon adding support for this.

@ad0861 @nviraj @soobbea Please let us know your preference on httr vs rcurl. We have started working on both. And it would be great to know which ones are more commonly used in the industry.

cc @LuisTejedaS @erfemega

I would recommend httr as it's very popular. The maintainers of the R library are also very active in the R community.

Thanks for taking this up @umeshp7! From my personal experience and working with other colleagues, httr seems to be more prevalent. In case it is useful, I am attaching stats for number of installs from RStudio's CRAN mirror (Not comprehensive as it is just a mirror but might indicate trends)
httr - []( URL
RCurl - []( URL

Just chiming in here that httr2 has been released. It is the successor to httr2. Not quite 300k a month, but having been released just over a year ago, ~35k/month is not insignificant.

[]( URL

I am seconding the call for httr2 code snipper support! It is more state-of-the-R-art because it is pipe-able and provides really good support for oauth flows and caching!


Question: Error with --snapshot-move-data Flag in Velero with Rook-Ceph Storage backup time?

@Lyndon-Li given that even "no nodes available" isn't necessarily permanent, I'm wondering whether the message-parsing approach might be too error-prone, missing edge cases, etc.

Error with --snapshot-move-data Flag in Velero with Rook-Ceph Storage backup time
Description
I am encountering an issue while attempting to back up and restore pods using Velero with Rook-Ceph as the storage backend. The error occurs when using the --snapshot-move-data flag for CSI snapshot data movement. Detailed steps and error messages are provided below.

Steps to Reproduce

- Setup Environment:
Velero configured with Rook-Ceph as the storage backend.
Kubernetes cluster managed by Kubespray.
- Backup Pod:
Execute the backup command with the --snapshot-move-data flag.
velero backup create media6 --snapshot-move-data
- Encounter Error:
Backup process fails

Current Issue
The backup processes fail with an error stating partially failed.

Expected Outcome
• Successful backup of pods using the --snapshot-move-data flag with Velero and Rook-Ceph.
• Successful restore of the backed-up data without encountering the DataUpload error.

Actual Outcome
• The backup process fails with an error related to partially failed.

Additional Information
• Current Environment:
- Kubernetes cluster managed by Kubespray.
- Velero for backup operations.
- Rook-Ceph as the storage backend.

Request for Assistance
I would appreciate guidance on the following:
1. Any additional configuration steps or best practices for using the --snapshot-move-data flag with Velero and Rook-Ceph.
2. Clarification on whether this is a known issue and if there are any patches or updates available to address it.

I have now installed the node-agent, and it is running. I attempted to back up, but I'm encountering a error.

Screenshot_752
Screenshot_749
Screenshot_751
Screenshot_750

Here is the related bundle file:
bundle-2024-08-06-17-38-42.tar.gz

@Lyndon-Li I don’t understand; there’s nothing related to my current issue.

Here is the error associated to the partially failed backup:
`"message": "found a dataupload velero/media6-fj8fl with expose error: Pod is unschedulable: 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. mark it as cancel",`

@Lyndon-Li You have no idea

`"message": "found a dataupload velero/media6-fj8fl with expose error: Pod is unschedulable: 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. mark it as cancel",`

 why this issue occurred, and the error message displayed does not provide a solution. The issue #7898 lacks any description for resolution. Is it possible that the problem lies in my installation steps or elsewhere? Any guidance to help resolve this issue would be appreciated.

There is no resolution/workaround, you have to wait the release of 1.14.1 or you need to downgrade to 1.13.x

@Lyndon-Li This means that the issue

`"message": "found a dataupload velero/media6-fj8fl with expose error: Pod is unschedulable: 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.. mark it as cancel"`,

did not occur in version 1.13.x, and my backup process was executed using the --snapshot-move-data flag, correct?

@Lyndon-Li Following your suggestion, I used version 1.13.2. After installing it and attempting to back up with the --snapshot-move-data flag, the backup status completed without any errors. However, the snapshot data was not moved as expected. I also checked the MinIO S3 bucket, and there was no Kopia folder created for the snapshot backup. In short, this is not meeting my requirements. Do you have any solutions related to this issue?

Screenshot_754
Screenshot_755
Screenshot_756
Screenshot_758
Screenshot_757
Screenshot_759
Screenshot_760
Screenshot_761

Here is the related bundle file:
bundle-2024-08-07-15-23-09.tar.gz

For Velero prior to 1.14, you need to add CSI plugin in installation if you want to use data mover feature.

@Lyndon-Li I'm encountering this error related to volume snapshots and classes, even though I've already defined them in my namespace.

Screenshot_762
Screenshot_763
Screenshot_764
Screenshot_765

Here is the related bundle file:
bundle-2024-08-07-17-03-56.tar.gz

@jayrajmeh0 You're using version 0.1.1 of the CSI plugin. For Velero 1.13 you need 0.7.x.

**@Lyndon-Li & @sseago Thank you both for your guidance and assistance with the backup process. I was able to take a snapshot backup using the flag --snapshot-move-data , which proves that I successfully completed this task.**

Screenshot_773
Screenshot_774
Screenshot_775
Screenshot_776
Screenshot_777
Screenshot_783

**The backup process has been successfully completed. However, when I removed the namespace associated with the backup shown in the screenshot above, I encountered an issue during my next task, which is restoring using Velero. The details of the issue are as follows:**

Screenshot_778
Screenshot_780
Screenshot_781

Here is the related bundle file:
bundle-2024-08-08-12-08-27.tar.gz

By checking the logs I found following error messages:

time="2024-08-08T06:25:44Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-08-08T06:25:53Z" level=error msg="fail to get DataDownload: fail to list DataDownload: etcdserver: leader changed" Action=PVCRestoreItemAction Namespace=velero OperationID=dd-4703bc72-cbef-4a31-a0bf-8a3ad433d90f.be3cfbe8-fe0a-41d7f03b9 cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/restore/pvc_action.go:240" pluginName=velero-plugin-for-csi
time="2024-08-08T06:25:53Z" level=info msg="Marking restore mongo-20240808115503 FinalizingPartiallyFailed" logSource="pkg/controller/restore_operations_controller.go:186" restore=velero/mongo-20240808115503

So it seems a random issue in etcd, if the problem is highly reproducible, you may want to check if there's any resource or perf issue in your cluster.

@reasonerjt
I have checked all the pods, and everything looks good without any errors during the restore process. Can you help me resolve the issue in more detail?

**@Lyndon-Li @sseago @reasonerjt**
**Additionally, I have another question related to Velero and Ceph storage. Currently, I have a three-node Kubernetes cluster with Ceph storage. If any of these three nodes were to be destroyed and my database backup is stored in the cloud using Velero, can I restore it to a new Kubernetes environment? What are the prerequisites for this, particularly concerning Ceph storage pools and OSDs? Do I need to set up the same names or configurations for Velero to restore everything correctly?**

@Lyndon-Li
**I successfully backed up one cluster and used the --snapshot-move-data flag to restore the backup on a second cluster. The restore process completed without any issues, and all statuses showed as successful. However, when I checked the MongoDB pod in the namespace of the new cluster, it was not functioning properly and encountered some issues.**

Screenshot_822
Screenshot_823
Screenshot_827
Screenshot_828
Screenshot_824
Screenshot_825

Here is the related bundle file:
bundle-2024-08-16-16-45-06.tar.gz

@jayrajmeh0

As for the latest issue, it seems the infrastructure where you restore the mongo DB does not meet the requirement of mongoDB.
I suggest you read the links in the error message to understand why it failed. IMO it is beyond the scope of velero.

Fail early DM pod checks have false positives
**What steps did you take and what happened:**
In Velero 1.13, we added code to fail early if the datamover pods are in an unrecoverable state: URL
In 1.14, some additional changes were made here: URL

In our OADP testing with Velero 1.14, we found that in some cases DDs or DUs were being canceled almost immediately. Digging into the code added in the above 2 PRs, there are a few situations where Velero considers a pod unrecoverable:
1) pod phase of Failed or Unknown
2) pod phase of Pending with an "unschedulable" condition
3) container status with ImagePullBackOff or ErrImgNeverPull

It was the second situation that was causing the failure for us. The pod will have an "unschedulable" condition until the PVC is is bound to a PV. In many cases, this all happens quickly enough that by the time Velero checks, the PVC is bound and the Pod is no longer unschedulable, but if the provisioner takes a few seconds to bind the PV and PVC together, the Pod may still have an "unschedulable" condition when the node agent first checks, which will cause the DU or DD to be immediately canceled.

We are seeing this happen frequently with Ceph volumes.

We have a couple of options here.

1) The simplest option would be to remove "unschedulable" from the list of unrecoverable conditions. As we see here, "unschedulable" does not always indicate an unrecoverable state. The only downside here is that we lose fast fail for permanently-unschedulable pods.

2) Slightly more complicated -- we could wrap the unschedulable check in a PollUntilContextTimeout call with some hard-coded timeout (2 minutes maybe?) -- this will result in almost-fast-failing for unschedulable pods. This assumes that no provisioner should take more than 2 minutes to provision the PV. That's probably safe.

I will be posting a draft PR for option 1) -- I'm open to moving to 2) if the consensus is that it's a better solution.

**Environment:**

- Velero version (use `velero version`): 1.14
- Velero features (use `velero client config get features`):
- Kubernetes version (use `kubectl version`):
- Kubernetes installer & version:
- Cloud provider or hardware configuration:
- OS (e.g. from `/etc/os-release`):

**Vote on this issue!**

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

- :+1: for "I would like to see this bug fixed as soon as possible"
- :-1: for "There are more important bugs to focus on right now"

@sseago

During data mover restore, I think this happens for almost every volume ---- the workload pod and PVC is firstly created, but the workload PVC is not bound to any PV until the DDCR completes, this won't complete in seconds/minutes. However, we didn't see this problem during the test.

Or do you mean this is an intermediate moment when the PV is provisioned and it takes some seconds to bound the PVC and PV.

It is definitely an intermediate scenario. I think some provisioners are fast enough that the code doesn't force cancel of DU/DD with the current code, but we have discovered with Ceph as a CSI provisioner, we are seeing frequent situations where DU or DD are getting canceled because the "unschedulable" pod state is considered "unrecoverable" by Velero even though it's not unrecoverable.

I personally think ignoring unschedulable makes the most sense.. an admin (or autoscaler) can scale up more nodes etc for scheduling to resolve. timing out at 1 minute and marking restore fail can be misleading. velero did everything right, infra was just not yet avail to schedule but could be totally fine 30 minutes later.

@Lyndon-Li here is the error we saw with Ceph:

 message: 'found a dataupload openshift-adp/backup20-llp79 with expose error: Pod
 is unschedulable: 0/6 nodes are available: pod has unbound immediate PersistentVolumeClaims.
 preemption: 0/6 nodes are available: 6 Preemption is not helpful for scheduling...
 mark it as cancel'

The "unbound immediate PersistentVolumeClaims" part is probably the most relevant part of the message. Unbound PVC is a temporary error.

@sseago
Then I think we can preserve the benefit by doing more check on the message though it is a little bit trick (when the message is changed by kubernetes, we also need to change), by this:

- We add a new parameter (e.g., `unscheduableMsg`) to `IsPodUnrecoverable`
- We check the Unscheduable container state only when `unscheduableMsg` is not empty
- When we detect the pod is Unschedulable status, we fruther check the message contains the `unscheduableMsg`
- There is only one place need to pass an non-empty `unscheduableMsg`, that is in `PeekExposed` where we set `unscheduableMsg` as `node affinity conflict`

What do you think?

cc @reasonerjt

@Lyndon-Li Looks like we have another bug seeing the same thing -- this time with node autoscaling -- i.e. no nodes available initially, but wait a little bit and new nodes are made available, but the DU has already been canceled: URL

All in all, I think the approach mentioned here is still valid since it is checking the string `node affinity conflict` and none of the existing issues have a container message like that.

On the other hand, I also agree this is a tricky approach. So the other way I suggest there is to simply remove the "UnReachable" container message check and let the DU/DD being cancelled after 30min timeout. This should be what #7899 is doing (if so we also need to change the design).

I am fine with the either way. @sseago Let me know your choice. And I will also discuss with others in the team.

We further discussed this issue and realized that there is no reliable way to filter out the unschedule problem caused by users' data mover node selection.
Therefore, we suggest to simply remove the changes remove the "UnReachable" container message check and let the DU/DD being cancelled after 30min timeout. This should be what URL is doing + data mover node selection design change.

@sseago Let us know your thoughts.

@Lyndon-Li so it sounds like you're saying the change in #7899 is appropriate? We still fail early for ImagePullBackOff or Failed pods, but for unschedulable, we just let the 30 minute timeout take effect? I'm marking that PR as ready for review now -- we can approve/merge that one or continue to discuss changes needed there.

Yes, #7899 has been merged. Meanwhile, I will submit a PR to change the node-selection design.

@Lyndon-Li reopening to consider this for 1.14.1 -- let me know if it's better to create a new issue instead.

IMO, cherrypicks don't have to reference a still opened issue, one could have cherrypicks that close an already closed issue for a different release branch.

No problem, we can use the same issue to track all (it was auto closed when merging #7899)

As the problem describe here, we cannot fail earlier, but we still need to preserve as more info as possible to help the troubleshooting when prepare timeout happens. Opened an issue #8125 for this.


Question: Feature requests - hotkeys for cycling spaces, mouse pointer warp with focus management?

This change shouldn't affect the performance.

The performance is the main goal of this project, I started it because Aerospace is too slow and there is no plan to fix it. See more here: #1

Yes, I use native hiding, Aerospace hides apps by hiding them in screen corners, but I think the main issue with Aerospace is that it checks all windows every time you try changing something which is inefficient but probably necessary to implement tiling management. This project doesn't need to care about windows placement so it's easier to make it faster.

Feature requests - hotkeys for cycling spaces, mouse pointer warp with focus management
Love this app and excited to see where it goes. Using multiple spaces is a great way to organize but the delay when switching (especially bad at 120hz) is nearly intolerable. It also exposed me to your Snippety app which I will also have to check out now.

I have a few suggestions, feel free to ignore if they do not fit with your plans for the app.

1. I am very used to using the built-in gesture to switch spaces (3/4 finger swipe left right when enabled in settings) and would love a way to assign shortcuts to cycle between the different "flash spaces" that I have created, like Ctrl+Left|Right does for the built-in spaces feature. I used BetterTouchTool to configure the 3-finger swipe left/right to toggle between two defined spaces, but it would be nice to have it switch between all of the ones I've created. I could probably make this work with BTT using a variable to keep track of the current space but it seems worth asking.

2. When switching app focus with the keyboard shortcuts it would be nice if the mouse pointer also moved to the focused application.

3. While I'm still adjusting to the auto-hiding of apps I occasionally feel like I "lose" an app (like the settings menu I've opened to tweak something) because I switched spaces. Maybe there could be a way to assign windows that get created in an existing space to a specific defined space when you switch rather than hiding from all spaces? OTOH I will probably get used to the behavior and this may be unnecessary long term.

Hi, thank you for the feedback!

Ad 1. I'm not sure if I understand it correctly. Would you like to have an option in the app to turn on the 3-fingers swipe to change spaces? Or would you like the app to expose some interface so that you can integrate another app that recognizes swipes? In any case, it sounds pretty cool and I'd like to add it.

Ad 2. Another great idea, I will try adding it.

Ad 3. Hmm, I'm not sure about this one. The key point is that I would like to keep the experience that FlashSpace is to help you but doesn't interfere too much with your actions. So if you call some app it appears as expected, as if FlashSpace didn't exist. Regarding auto assignments, I'm worried about ending up like most tiling managers with everything blinking and jumping when a new window appears and gets assigned somewhere, especially small windows :D. However, I understand the point here, I'll think about it.

Thanks for valuable ideas 🍻

For #1 just keyboard shortcuts to cycle spaces is fine and keeps this tool keyboard focused. BetterTouchTool is already great at binding a gesture to a keyboard shortcut so it makes sense to keep those separate.

For #3 I do agree it starts to get into full on window management which does seem out of scope. I think this was just onboarding, after a few hours I am pretty used to how it works already.

Hi! I added the features you asked for. I really love both, great suggestions! 🍻

Regarding the cursor warp, I added two modes separately. One is for hotkeys changing focus and one is for hotkeys changing workspaces.

Question about: The last app on the list will be focused when switching to the workspace.
Is there any plan in mind to make it configurable?

Let's say: When changing to another workspace, it should focus not the last app on the configured list of the workspace, but actually the last focused app in that workspace.

Interesting, definitely worth adding as an option. I plan to add settings, so I could probably add such feature there.

Is this behavior if implemented can somehow affect the performance? I know it is hard to say, but I tested and comparing to AeroSpace, I can see FlashSpace changes faster even in the v0.2.3 version (early project development).

Are you relying in different APIs than AeroSpace to the core of its APP? I know it is hard to compare as the tools try to solve different problems (AeroSpace adds window management behavior, but I am not interested in that). I ask that, because I couldn't really find quickly these infos and I am not Swift Developer to understand the code basis in a hurry, I'd need to take a time to onboard, so asking seems easier.

Really enjoyed the app UNIX philosophy (not sure if it was intended): do 1 thing and do it the best way possible: Workspaces management. I am not interested in window tiling and I can see how AeroSpace is interested in that, but not my use case. Thanks for the overview. I can't wait this project to grow :-)


Question: Add small caps æ and ø?

Stop being such a rude asshole to Owen.

He did fix it.

image

It's possible he didn't update the TTF's distributed, or for some other reason, you're still using an old font. Making a duplicate issue instead of replying in the existing one with something nicer like "it still doesn't work for me, any idea why?" will get you a lot more traction.

Closed.

Add small caps æ and ø
If I download Besley* from URL I get an old version (1.8.3) which is useless in Norwegian because of the ø's. If I download from github I get version 2.000 which is equally useless for the same reason. But if I download from Google fonts I get version 2.001 which is perfect! This version is called Besley (without an asterisk) and contains no weight called fatface – but a «black» which looks very similiar. Will you update the fonts – at least in indestructibletype.com?

@sigurdhasle opened #32 which was quite rude and also a duplicate so I closed this but you forgot to add the substitutions to the lookup. I'll open a PR for it.

I am sorry if you consider me a rude asshole. I opened a new issue because I was not aware that I could comment in a closed one. I am just frustrated that I cannot use the small caps: it still doesn't work for me. Do you have any idea why?

I have downloaded and installed the files again today – they should not be old files.. The small caps Æ still looks like a lowercase æ – both in MainType and LibreOffice.

Add Æ and correct Ø to small caps
Wow! You fixed it – NOT! The small caps still need a bit of work. The æ's are _still_ just lowercase instead of small uppercase letters. The ø's should be the same shape and size as the o's, but with a slash through it. The slash should go throught the middle of the ring (the two crescents should be the same size). Slurv!

image

I am sorry if you consider me a rude asshole. I opened a new issue because I was not aware that I could comment in a closed one. I am just frustrated that I cannot use the small caps.
I have downloaded and installed the files again today. The small caps Æ still looks like a lowercase æ.


Question: Update min LibCST version when it supports Python 3.9 relaxed decorator syntax?

Progress update: I have a working CI job that produces binary wheels for LibCST with the new rust-based parser for any combination of python (3.6, 3.7, 3.8, 3.9, 3.10) & (macos x86_64, macos arm64, 32bit linux, 64bit linux, 32bit windows, 64bit windows).

Example artifacts at URL (it's a zipfile with wheels in it).
You can `pip install` the relevant wheel from the zip on the above platforms and run

LIBCST_PARSER_TYPE=native python -c 'import libcst; print(libcst.parse_module("foo(a, b)"))'

To see the new parser in action. Note: match statement is not implemented yet in these.

Update min LibCST version when it supports Python 3.9 relaxed decorator syntax
python
with (a() as b, c() as d):
 pass

@(lambda f: f)
def g():
 pass

is valid Python 3.9 syntax which triggers URL and, in `--refactor` mode, URL and URL Once these upstream bugs are fixed, we should:

- [x] update the minimum version of Black
- [x] update the minimum version of LibCST

Per URL an upcoming release will enable their new Rust-based parser for 3.10 & 3.11 grammar by default :tada:

(maybe this will also make it faster?)

`black` also now supports 3.10+ syntax with patma / parenthesized with statements.

Black now works, so we're just waiting for enabled-by-default `native` parser in libCST :tada:

This all works now; we just need to add some tests and bump the minimum required version.

I tried adding a test to check for python 3.9 syntax but apparently libcst has not yet added syntax checking for this python version. python39syntax

I tried adding a test to check for python 3.9 syntax but apparently libcst has not yet added syntax checking for this python version. URL

I tried adding adding a test to check for python 3.9 syntax but it failed. Apparently libcst has not yet added syntax checking for this python version. URL

3.10+ support
PEP 617 (if it is accepted, and it probably will) is out and I'm wondering if LibCST's underlying parser is capable of parsing PEG grammar. With 3.10, the LL(1) restriction on the grammar will be deferred and this means that `lib2to3.pgen2` won't be able to parse new changes on the python grammar. I'm not sure about internals of LibCST but from what I have seen in readme that it uses something that bases on `lib2to3.pgen2`. Does LibCST will continue to support newer python versions and their grammar? (We are currently using lib2to3 as our refactoring tool on unimport but we might need to migrate another tool to support 3.10+ which is why I am asking)

I haven't discussed with the other maintainers yet, but here are some thoughts:

1) My reading of the PEP is that it specifies that no LL(1)-incompatible change will be made to the language grammar until _after_ 3.10 is released. That gives us some time to adapt and should mean that supporting both 3.9 and 3.10 with the current LibCST parser will be straightforward, so this issue should probably be titled "3.11+ support."

2) In the long run I think we would want/need to rebase LibCST on top of a PEG parser in order to support Python grammar post-3.10. We will need to see how much of the code of the PEP 617 parser we can reuse for LibCST, given that we need syntactic trivia (whitespace, comments) and Python doesn't.

Oops, looks like I mis-read the PEP. It seems the old parser will be removed in 3.10 and non-LL(1) constructs may be added then. So we will need to address this in order to support Python 3.10.

Do you know a good PEG parser for python that also constructs a CST, I'm not much familiar with PEG :/

No, I doubt any such thing exists. We, or someone, would have to build it. I just sent a message to the python-dev thread about PEP 617 inquiring about the potential for reusing some of pegen (the new parser in PEP 617) for this: URL

It seems like one option would be to use URL as a starting point for a new Python 3.10 compatible parser.

Yes, that would be a great idea. If there is anything I can help you folks, just let me know.

I'm thinking about augmenting stdlib ast module to have some lib2to3 features (because lib2to3 is going away). If this is of interest to you:
 URL
(No promises that I'll do the work of course, but I have high hopes)

This issue is a blocker for us:
- We are trying to integrate `libcst` into `wemake-python-styleguide` which supports all versions from 3.6
- We even have a PR ready: URL
- But, since 3.9+ is not supported, we have to reconsider this idea
- The same is true for several other projects of mine which potentially could use `libcst` for code auto-formatting 😞

Are there any updates? Is there anything I can help with?

No updates yet apart from some of us hacking around on our spare time to get LibCST onto a peg parser. I'll update here as soon as work starts on this in earnest. I'm hoping for good news in a month or so.

Good news! :) I'm working on this as part of my main job now. The current status is: in my branch there's a rust implementation of a peg parser for a small part of the python grammar, but complete with whitespace handling (thanks to the awesome work on whitespace parsing and tokenization by @bgw). It can successfully roundtrip simple functions like this without any extra string allocations (the built CST shares memory with the input source string), and it seems to be _real_ fast (it will be interesting to see if this holds as I add more and more of the grammar).

I've opted for implementing the heavy lifting in Rust, and then expose the resulting CST as Python objects as part of a relatively high level interface that will be compatible with URL
I'm hoping this will let us get significant speedups in both CPU and memory usage while being memory-safe.

There's still lots of work to do, for example:
- better error messages
- comprehensive testing
- more grammar
- wrap the API to be usable from Python

I'll keep updating this issue as I go along

Here's the current status of the rewrite: I think I have 99% of the Python 3.8 grammar implemented, and the parser can roundtrip (parse -> serialize losslessly; i.e. input bytes are same as output bytes) all of the LibCST python implementation (i.e. this repo). This is a big milestone but there's still some work to be done. But first, some details in case you want to play around:

The code currently lives still in URL you should be able to `cargo build` in that directory to get a debug build. This will put a binary called `parser` (`parser.exe` on Windows) in `target/debug/` which
* accepts python code on stdin, then parses it into a CST and serializes it back onto stdout
* optionally (when given the `-d` flag) dumps the internal CST representation before the generated source - this should look familiar if you've used `python -m libcst.tool print` before
* if the first argument is `-n` it doesn't output anything (except parse errors), just parses the input

Here's what I know still needs to be done:
* [ ] wrap the API in a thin python layer to be usable from LibCST
* [ ] hunt down bugs in the grammar by parsing all the python source code ever
* [ ] implement missing grammar features like match statement, walrus, or parenthesis around `with` statements
* [ ] there's a bug with whitespace parsing around the end of multiple nested indented blocks; this results in duplicate whitespace in certain situations - but not for any of the LibCST source code apparently
* [ ] figure out how to make a LibCST release that can switch between the new and old parser backends
* [ ] CI for the rust code

And then on top of this it'd be nice to look into:
* [ ] Perf optimizations. I haven't done any :)
* [ ] Make the parser error-recoverable; that is, for a syntactically invalid Python document, try and produce _something_ that resembles a CST and is usable. This would be great for interactive use cases like IDE linting.
* [ ] Improve the parser error messages. They are really basic at the moment, just outputting expected characters at the furthest parsed position.
 * [ ] There has been some great work in CPython recently around this front, and we should be able to port most of it over easily, if rust-peg can be enhanced to support early termination

I've setup an example using github actions here: URL

Sample run: URL

Hi, What is the advice for libraries that depend on this project? Can we expect 3.10 support in the near future of should people start looking for alternatives? It seems like Black has solved this with their lib2to3 fork.

I plan to release a version with opt-in 3.10 support by the end of the year. If that goes well, a new release in January will have 3.10 support by default.

#566 proposes to merge the Rust-based parser. After merging I'll follow up with (much simpler) PRs to implement match statement, and parenthesized context managers.

Note that for 3.10 (and 3.11) support, you still need to set `LIBCST_PARSER_TYPE=native` env var for now. We'll be removing this restriction in a future release (soon (tm)).

- It's been a while - any plan for when the native parser will become the default?

- I've discovered a fair few bugs while using it, and could take a day to do a more thorough search if there are resources to fix them. What's the maintenance status of libcst, and is this worth trying?

- Unsure on when rust parser will become the default, but `export LIBCST_PARSER_TYPE=native` is ok for now.
- I haven't faced any bugs in practice with libcst on the latest release. Can you mention what bugs? Have you created github issues for them?

- Yes, I have a workaround you can see in #917, but it's pretty ugly.
- URL includes seven open bugs in the current version, all of which I hit in practice. In the last few months there's also #829, #846, #854, and #916.

#929 is landing in a moment, after which I'll release it with a major version bump 🎉


Question: Create a bidscrambler for EEG data in the BrainVision Core format?

Hi @schoffelen could you test the implemented scrambling of the BrainVision format EEG data? Starting from the original dataset of 2.4, you can use URL to make a BrainVision formatted version for some of the subjects, and then apply BIDScramble. Following scrambling of the EEG data (with the option `brainvision`, the code for that is in URL the EEG time series should contain random noise. It seems to work in my hands, so I hope that testing for you will be easy.

Create a bidscrambler for EEG data in the BrainVision Core format
The BrainVision Core file format is one of the file formats for EEG data supported in BIDS. Compared to the EEGLAB file format (which is also supported), it is much simpler to read and write. Hence I propose that we first implement a scrambler for this format.

One EEG recording consists of an ascii `.vhdr` (header) and `.vmrk` (marker/events/trigger) file, plus a data file with the actual EEG data. The data file often has the extension `.eeg`, but that extension is not required. The data file name (including extension) is specified _inside_ the header file.

Both header and marker file resemble the windows-ini and the TOML file format.

In principle the data file can be either ASCII or binary, and either serialized or multiplexed (and all those files exist in the wild), but the formal 1.0 specification PDF prescribes that it must be multiplexed binary (which makes things much simpler).

To remind myself: MNE python relies on URL for their (somewhat limited, i.e. it does not support a round trip in reading and writing) Brainvision IO

Note that this issue was duplicated in #31 and has been implemented with pull request #32

Implement scrambling of EEG data for use case 2.4
@marcelzwiers @schoffelen @arnodelorme @CPernet

For your information, I am working on the implementation of the BrainVision Core data format in the BIDScramble. The EEGLAB `.set` format is too complex for now, but the data in use case 2.4 can be converted (using FieldTrip) from EEGLAB to BrainVision format without compromising the analysis pipeline.

As the [pybv]( URL reader was a bit of a mess with regards to dependencies and not symmetric in reading and writing, I implemented my own BrainVision read and write package. See URL I will make sure that this gets added to pypi to allow installation with pip.

The BrainVision code is now on PyPi at URL

I implemented an automatic release "action" following the instructions at URL

As discussed with @CPernet we need to come up with a few thought-through meaningful scrambling options for MEEG data, beyond time series scrambling. Let's reopen this issue to bounce ideas back and forth.

We probably need to have 2 or 3 additional ways to break the spatio/temporal/spectral structure in the data.

I am currently thinking about a combination of spatial filtering and blockwise scrambling. For ERP'ish application this probably also would need to take the structure of the events into account.

Specifically, using spatial unmixing (based on 'some' heuristic), then a scrambling step (e.g. scrambling the order of the latent components and/or blockwise scrambling (i.e. keeping the overall spectral structure in the data, and/or time-locked to event stuff), then again a spatial mixing step, would be a flexible framework that allows for a mixture of spatiotemporal scrambling, preserving aspects of the overall structure in the data

I agree that within the ICA framework, one is likely able to mess stuff up different ways (after characterizing the evoked and spectral aspects id IC). I also agree (as discussed on line) that often the pipeline will be something like 'can want to see if the pipeline can detect X', so the feature of interest needs somehow to be present.

- one could play with the weight matrix ?? maybe as a way to 'remove' known sensitive attributes (not something I'd do by default, but if a data control right holder indicates an aspect that needs nullifying, this can be a way)
- specific IC could be made noisier adding pink noise or a sin where evoked response is (say found IC with spectral / evoked response, ie increase energy following events anytime from 50 to 1000ms, no need to be specific IMO -then add, it exist tools for simulations I used that can be leveraged for that)
- total randomness, chunk time courses based on autocorrelation size (to keep main spectral, autocorrelation) and exchange within/between IC and remix

not sure all makes sense, but putting some ideas down ...

I can confirm that the scrambler for brainvision data works.

The current default is nulling, but when I add the
permute
 as a method, the data are nicely permuted, confirmed by visualizing the time series of a given channel, and by comparing the histograms of the amplitudes of the scrambled and non-scrambled versions.

One additional observation I had, is that I need to specify
cfg.readbids='no'
 in order to load the scrambled data into matlab (using
ft_preprocessing
). The reason for this seems to be that that scrambling (I did a two step approach with:

scramble in out stub
scramble in out brainvisino permute

resulted in a dataset with empty run-specific json files, causing fieldtrip to choke)

Would those be the `eeg.json` files that are empty? I believe that empty run-specific json files are not BIDS compliant, which means that BIDScramble has a bug.

Considering my previous comment: upon rereading the documentation I suspect that scramble should also have been called with the `json` action. The `stub` action creates the empty files, the `json` action allows copying json file content over (with or without modifications).

The use case specific scramble instructions should be updated/completed/reviewed. I'll make a new issue for that.

Is there any documentation on empty json-files being non-compliant? Because I think there is nothing wrong with that (it's not desirable, but also not wrong))

Yes, as I wrote in this comment I also think it is a user error. Some of the json files should have been copied over with `--preserve`, as those are needed for processing.

I am surprised that you write that scrambled data can never be BIDS compliant. Why not? In general - with the proper options - I think that the scrambled data would onl;y have different numbers, not a different structure. If not compliant, I don't think that strict BIDS apps like MRIQC will even work on the scrambled data.


Question: DenoisedPathTracer does not appear as an option in the Renderer drop-down menu?

The README that you read is for a WIP newer version of the plugin that is not yet released. The `DenoisedPathTracer` is not in the 0.4.0 version of the plugin. This README is for the 0.4.0 version of the Denoiser plugin.

DenoisedPathTracer does not appear as an option in the Renderer drop-down menu
As stated. It is just not there. I got two other options though - ALBEDO and NORMAL. I have no idea what to do with this.

Unline the member in question, I do not get denoised images in my snapshots folder.

Have you downloaded OIDN and set it up according to the instructions?
 URL

If so, have you rendered to the Target SPP?

Sorry, my bad, I just looked some more and it is there. Just very confused about the whole thing.

`DenoisedPathTracer` missing from Advanced tab
The `ALBEDO` and `NORMAL` renderers show up alongside the default `PathTracingRenderer`, but `DenoisedPathTracer` (mentioned in the readme) is nowhere to be seen.

obrazek

However, denoising still works (probably) as intended, since the render process starts out by first rendering the normal and albedo passes before anything else, and the denoised image *can* be found in the scene's `snapshots` folder:

obrazek

- Chunky version: 2.4.4
- Denoiser Plugin version: 0.4.0
- OIDN package: oidn-1.4.3.x64.vc14.windows
- Java version: 17.0.1.12 (Adoptium JDK)
- JavaFX SDK version: 18.0.2
- OS: Windows 10 Home x64 v1909 (build 18363.1556)


Question: FkProducer.java:12: Fake Producer implementation?

@h1alexbel this is your task now, please go ahead. Deadline (when this ticket should be closed) is ``2023-06-22T20:09:04.176448``.

Estimation here is ``60 minutes``, that's how much you will be paid.

Remember, you don't have to solve everything in this ticket - you can solve it partially and leave ``todo`` markers in the code, which will become future tasks.

If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.

If you don't have time or simply don't want to work on this, you can always resign.

FkProducer.java:12: Fake Producer implementation
The puzzle `10-404e50f4` from #10 has to be resolved:

 URL

The puzzle was created by @h1alexbel on 24-Feb-23.

Estimate: 45 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be \"done\" when the problem is fixed and the text of the puzzle is _removed_ from the source code. Here is more about PDD and about me.

The fake producer can be built using an in-memory XML sheet.

xml

 1
 org.apache.kafka.common.serialization.StringSerializer
 org.apache.kafka.common.serialization.StringSerializer

 1

 name
 1
 key-0

 2

 name2
 3
 key-1

On each
send
 method execution new chunk will be added. But where to store this file with initial values? So far, I don't know

@0pdd 2 puzzles #292, #293 are still not solved; solved: #364.

@0pdd the puzzle #373 is still not solved; solved: #292, #293, #364.

@0pdd all 4 puzzles are solved here: #292, #293, #364, #373.

FkBroker.java:29-31: Broker has too many methods We...
The puzzle `292-fab4489f` from #292 has to be resolved:

 URL

The puzzle was created by @h1alexbel on 12-Jun-23.

Estimate: 60 minutes, role: DEV.

If you have any technical questions, don't ask me, submit new tickets instead. The task will be \"done\" when the problem is fixed and the text of the puzzle is _removed_ from the source code. Here is more about PDD and about me.

@0pdd thank you for reporting this. I'll assign someone to take care of it soon.

@l3r8yJ this is your task now, please go ahead. Deadline (when this ticket should be closed) is ``2023-06-22T16:19:05.771309``.

Estimation here is ``60 minutes``, that's how much you will be paid.

Remember, you don't have to solve everything in this ticket - you can solve it partially and leave ``todo`` markers in the code, which will become future tasks.

If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.

If you don't have time or simply don't want to work on this, you can always resign.

@h1alexbel ok, I've removed this task from scope. I'm not managing it anymore.

@h1alexbel I've just registered this ticket as a task and will assign it to someone soon. Thanks!


Question: switch-input-source-backward need to be handled as well, in addition to switch-input-source?

I can't reproduce :-(
At least not when tested on _Wayland_.
Also tested with AND without a custom key bound to the switching.

switch-input-source-backward need to be handled as well, in addition to switch-input-source
1) The "switch-input-source-backward" binding should be handled in addition to "switch-input-source".

You can see it using next command line output:

 gsettings list-recursively org.gnome.desktop.wm.keybindings|grep switch-input

By default Super+Space switches layout forward, while Super+Shift+Space switches it backward.
Suppressing of the popup window works for "forward" switching at the moment, while Super+Shift+Space still shows the popup.

It also can be reproduced using next command line settings for Ctrl+Shift switch:

 gsettings set org.gnome.desktop.wm.keybindings switch-input-source "[' Ctrl_L']"
 gsettings set org.gnome.desktop.wm.keybindings switch-input-source-backward "[' Shift_L']"

or next ones for Alt+Shift keystroke:

 gsettings set org.gnome.desktop.wm.keybindings switch-input-source "[' Alt_L']"
 gsettings set org.gnome.desktop.wm.keybindings switch-input-source-backward "[' Shift_L']"

That way Ctrl+Shift or Alt+Shift keystroke will act just like it works in Windows, but the popup window will still appear for each second Ctrl+Shift or Alt+Shift stroke from time to time (not each time though).

2) Also it would be nice to integrate that Ctrl_L and Shift_L (or Alt_L and Shift_L) hack somewhere in plugin settings as option, because it's not obvious how to obtain Windows like keyboard layout switching "out of the box".
The default Super+Space combination is not so "quick" as for me.

1. I'm not surprised that i hadn't handled also the shift(backwards) case, i didn't know this command existed. But I'm surprised that no one ever reported it.
2. Will fix it.

3. I don't know the Ctrl_L hacks? Can you give me some hints, or an merge request?

1. I suppose that the backward switch is rarely (never) used because most of users just cycle between native and English language, it's enough to press Super+Space for that
2. TA in advance
3. In short - it's not possible to assign Ctrl+Shift or Alt+Shift as language switch sequence using Gnome Settings tool.
Meanwhile it's possible to do that using Gnome Tweaks Utility (it has Keyboard & Mouse > Additional Layout options > Switching to another layout > [ ] Ctrl+Shift checkbox).
It will work, though it will be not possible to see changing in status bar icon, it will always show 'en' (Centos Stream 9 / Gnome Standard Wayland, other distros also seems are affected).

As workaround it's possible to execute the next commands from command line (I mentioned them above), after that keyboard layout switch will work as expected and the icon in status bar will change accordingly:

 gsettings set org.gnome.desktop.wm.keybindings switch-input-source "[' Ctrl_L']"
 gsettings set org.gnome.desktop.wm.keybindings switch-input-source-backward "[' Shift_L']"

I initially found the gsettings hack here URL
sorry for the Russian text, but it could be autotranslated using Google Chrome Translate option.

Also next link might explain the same issue better:
 URL

That way I found existence of switch-input-source-backward and faced with the popup which had to be suppressed by your plugin.

So it will be great to create small setup dialog for your plugin which will allow to choose proper settings for language switcher (default Super+Space, Ctrl+Shift, Alt+Shift or PrtSc/SysRq) .
I would raise PR from my side though I never developed the GNOME extensions, it will take some time to deep-in there.

I've made a quick patch adding new const for "switch-input-source-backward" shortcut and duplicating code for ""switch-input-source". I didn't add "disable" part because of issue #5

--- extension.js
+++ extension.js
@@ -26,6 +26,7 @@
 const { Gio, Meta, Shell } = imports.gi
 const Main = imports.ui.main;
 const SWITCH_SHORTCUT_NAME = 'switch-input-source'
+const SWITCH_BACK_SHORTCUT_NAME = 'switch-input-source-backward'
 const InputSourceManager = imports.ui.status.keyboard.getInputSourceManager();

 class Extension {
@@ -35,10 +36,16 @@
 enable() {
 log(`enabling extension '${Me.metadata.name}': bypassing language switcher popup.`);
 Main.wm.removeKeybinding(SWITCH_SHORTCUT_NAME);
+ Main.wm.removeKeybinding(SWITCH_BACK_SHORTCUT_NAME);
 Main.wm.addKeybinding(SWITCH_SHORTCUT_NAME,
 new Gio.Settings({ schema_id: "org.gnome.desktop.wm.keybindings" }),
 Meta.KeyBindingFlags.NONE,
 Shell.ActionMode.ALL,
+ this._quickSwitch);
+ Main.wm.addKeybinding(SWITCH_BACK_SHORTCUT_NAME,
+ new Gio.Settings({ schema_id: "org.gnome.desktop.wm.keybindings" }),
+ Meta.KeyBindingFlags.NONE,
+ Shell.ActionMode.ALL,
 this._quickSwitch);
 }

To apply this patch you need to perform such actions in a Terminal
1) Go to extension directory
`cd "~/.local/share/gnome-shell/extensions/quick-lang-switch@ankostis.gmail.com/"`
2) Create patch file
`nano disable--switch-input-source-backwards.patch`
3) Paste content of the patch (above) with right mouse click (or Ctrl+Shift+V) and save it with Ctrl+S
3) Apply patch (and backup original file as extension.js.orig)
`patch -b < disable--switch-input-source-backwards.patch`

Switcher popup breaks after disabling the extension (needs gnome-shell restart)
OS: Ubuntu 22.04.1
GNOME: 42.5
Session: X11

Steps to reproduce (with default key-binding + Space):
1) Enable, then disable extension without gnome-shell restart
2) Try to hold "Super" and press "Space" multiple times

Expected result:
While you're holding "Super" pop-up remains on the screen and you can switch languages with "Space"

Actual Result:
Pop-up disappears immediately after you've pressed "Space" the second time.


Question: J2objc 3.0.0 and JDK 17?

That's tough to answer with a yes or no, as there are really three questions with this one:

1. Does j2objc support all Java 17 language features?
2. Does j2objc support new Java 17 JRE API?
3. Does j2objc run on Java 17?

1) AFAIK, j2objc supports the latest language features, like text blocks and switch expression statements. Much of that support is free, because j2objc uses javac as its front-end; for example, javac converts text blocks into regular strings, so j2objc didn't need to be updated to support that feature.

2) No, J2ObjC's JRE is a fork of Android's libcore library (its JRE), so if it's not available in Android, it's not available on iOS. That's intentional, as the purpose of J2ObjC is to support shared Java across an app's clients, not to be a complete Java platform (that's why we don't/can't call it "Java").

3) No, because the javac team (see 1)) keeps changing access to its packages. So it works on Java 11, but it is currently broken on Java 17 until someone fixes the necessary flags (contributions welcome!).

The Android Gradle Plugin should be able to run command-line tools like j2objc in a separate shell (and likely already does). So try installing both Java 17 and Java 11 on your system, with the j2objc action defined by the flags in the j2objc.sh script. That script is also where the flags aren't correct for Java 17, so if you can fix them, great!

J2objc 3.0.0 and JDK 17
Hello J2OBJC team,
I am new to J2OBJC. I saw that there was version 3.0 but it seems it is not an official release. I have two questions below:
1. Does j2objc 3.0.0 support Java JDK 17?
2. When will there be an official release?

Developing and maintaining J2OBJC is a great thing, thank you for your effort. I am looking forward to your response.

We stopped distributing release bundles for two reasons. First, with the new Apple-related architectures and added JRE features, release bundles have been too big to upload to GitHub for years now. Second, developers want the latest features and bug fixes within days of their being committed to our repo, so most projects build j2objc from HEAD. We realized that j2objc is best suited for large, multi-platform app builds, and those teams wisely demand full control over what's in the apps. So between customer priorities and GitHub limits, releases no longer make sense.

The reason for the 3.0.0 tag is because we distribute the annotations jar via Maven, since Google Guava depends upon it. The last time we added an annotation that the Guava team used, we had to bump up our jar's version number.

No, but our latest source builds with JDK 17. However, our biggest clients are still on Java 11, so it hasn't been tested as thoroughly. You can use post-Java11 features that are implemented by javac, like switch expressions, text blocks, and pattern matching for instanceof. Features requiring JRE updates, like Records, are not yet supported.

@tomball: Good to see that J2ObjC now builds with JDK 17 👍 I'll try it soon and update #2095 and the Requirements section of the root-level README file if it works.

FYI, I'm still working on support for switch expressions. With the Java 21
added support for patterns, the ASTs javac returns are different between
Java 17 and 21.

On Thu, Feb 6, 2025 at 1:27 AM Adil Hussain ***@***.***>
wrote:

Is Java 17 supported?
This project's README file specifies "JDK 1.8 or JDK 11" in the Requirements.

Lines 191 to 205 of the make/common.mk file suggest that Java versions 12 to 19 are supported.

Can I build and use J2ObjC with Java 17? I ask because the latest Android tooling (specifically, the Android Gradle Plugin) require Java 17 or later. (For reference, see the Android Gradle Plugin 8.0.0 release notes.)

Thanks for the answer 👍 We'll try it out in the coming weeks and see how we get on with it.

I just re-read my question and @tomball's answer and I see that I was unclear about what I was asking exactly.

What I meant to ask was: Can I build J2ObjC with my `JAVA_HOME` environment variable set to JDK 17? I just tried it and the answer right now is: no. The build fails for me with the following error:

shell
...
building cycle_finder jar
building jre_emul archs: iphone64 simulator64 simulator
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
translating jre_emul sources
install -m 0644 build_result/cycle_finder.jar ../dist/lib/cycle_finder.jar
error: internal error translating "/Users/adil/Work/j2objc/jre_emul/openjdk/src/macosx/classes/sun/nio/fs/MacOSXNativeDispatcher.java"
java.lang.IllegalAccessError: class com.google.devtools.j2objc.javac.TreeConverter$1 (in unnamed module @0x1a708820) cannot access class com.sun.tools.javac.parser.Tokens$Comment$CommentStyle (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.parser to unnamed module @0x1a708820
 at com.google.devtools.j2objc.javac.TreeConverter$1. (TreeConverter.java:1477)
 at com.google.devtools.j2objc.javac.TreeConverter.convertInner(TreeConverter.java:280)
 at com.google.devtools.j2objc.javac.TreeConverter.convert(TreeConverter.java:261)
 at com.google.devtools.j2objc.javac.TreeConverter.convertCompilationUnit(TreeConverter.java:232)
 at com.google.devtools.j2objc.javac.JavacParser.parseFiles(JavacParser.java:212)
 at com.google.devtools.j2objc.pipeline.FileProcessor.processBatch(FileProcessor.java:137)
 at com.google.devtools.j2objc.pipeline.FileProcessor.processInputs(FileProcessor.java:67)
 at com.google.devtools.j2objc.pipeline.TranslationProcessor.processInputs(TranslationProcessor.java:84)
 at com.google.devtools.j2objc.J2ObjC.run(J2ObjC.java:127)
 at com.google.devtools.j2objc.J2ObjC.main(J2ObjC.java:176)

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "com.google.devtools.j2objc.ast.CompilationUnit.getSourceFilePath()" because " " is null
 at com.google.devtools.j2objc.javac.JavacParser.parseFiles(JavacParser.java:214)
 at com.google.devtools.j2objc.pipeline.FileProcessor.processBatch(FileProcessor.java:137)
 at com.google.devtools.j2objc.pipeline.FileProcessor.processInputs(FileProcessor.java:67)
 at com.google.devtools.j2objc.pipeline.TranslationProcessor.processInputs(TranslationProcessor.java:84)
 at com.google.devtools.j2objc.J2ObjC.run(J2ObjC.java:127)
 at com.google.devtools.j2objc.J2ObjC.main(J2ObjC.java:176)
make[1]: *** [/Users/adil/Work/j2objc/jre_emul/build_result/Classes/.translate_mark_jre_emul] Error 1
make: *** [jre_emul_dist] Error 2

This is not a blocker for me. Taking @tomball's answer into consideration, I have both JDK 11 and 17 installed on my machine. I have my `JAVA_HOME` environment variable set to JDK 17 except in a small number of isolated processes where I change it to JDK 11. For example, when building J2ObjC and when running the `j2objc` command.

Still working on switch expressions. The basics are improving with each commit, but the language spec got a lot more complicated in Java 21 with pattern and guard support.

Only as requested by customers, and then only APIs that are implemented on Android and don't require massive changes. Examples of new APIs that won't be supported include lightweight threads, pseudo-random number generators, foreign memory access API, and context-specific deserialization filters. Generally, j2objc's purpose is to support code-sharing of Java on iOS that also runs on Android or is transpiled via J2CL on web platforms.

Yes, j2objc is currently in use inside Google on Java 22, and I've been testing new language features on Java 14, 17, and 21.


Question: docs: Add a mention for how to glob image files (frontend bundler)?

I looked into this a bit. For my own reference, here's a more concrete example:

* `entry.js`:

js
 console.log(import("./academic-cap.svg"))

* `academic-cap.svg`:

svg

When the above files are bundled with `esbuild entry.js --bundle --format=esm --splitting --outdir=out --metafile=meta.json --loader:.svg=file`, I get the following `meta.json` file:

json
{
 "inputs": {
 "academic-cap.svg": {
 "bytes": 7,
 "imports": []
 },
 "entry.js": {
 "bytes": 42,
 "imports": [
 {
 "path": "academic-cap.svg",
 "kind": "dynamic-import"
 }
 ]
 }
 },
 "outputs": {
 "out/entry.js": {
 "imports": [
 {
 "path": "out/academic-cap-JHXDOPWG.js",
 "kind": "dynamic-import"
 }
 ],
 "exports": [],
 "entryPoint": "entry.js",
 "inputs": {
 "entry.js": {
 "bytesInOutput": 50
 }
 },
 "bytes": 63
 },
 "out/academic-cap-55CCFTCE.svg": {
 "imports": [],
 "exports": [],
 "inputs": {
 "academic-cap.svg": {
 "bytesInOutput": 7
 }
 },
 "bytes": 7
 },
 "out/academic-cap-JHXDOPWG.js": {
 "imports": [],
 "exports": [
 "default"
 ],
 "entryPoint": "academic-cap.svg",
 "inputs": {
 "academic-cap.svg": {
 "bytesInOutput": 56
 }
 },
 "bytes": 124
 }
 }
}

This information lets you map from `out/entry.js` => `out/academic-cap-JHXDOPWG.js` => `academic-cap.svg` but there is no link to `out/academic-cap-55CCFTCE.svg` (the file referenced by the imported string).

docs: Add a mention for how to glob image files (frontend bundler)
Currently it seems that if you want to use `webpack_path` with an image in `frontend/images/`, it's not possible if the same image isn't used in a JS or CSS file (e.g. `import "../images/file.png"` in `frontend/javascript/index.js`).

This limitation isn't mentioned anywhere in the docs, as far as I can see.

**Bridgetown Version**: 0.21.5

**To Reproduce**

1. Create a fresh Bridgetown site
2. Add an image to `frontend/images/`
3. Try to use that file on a liquid page (e.g. ` `)

**Current behavior**

An opaque error is printed to the logs:

[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.
[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.
[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.
[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.
[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.
[Bridgetown] Webpack: There was an error parsing your images/favicon.png file. Please check your images/favicon.png file for any errors.

And the emitted mark-up is:

**Expected behavior**

I would expect, based on the documentation, to be able to use any image from `frontend/` with the `#webpack_path` helper to take advantage of file hashes etc…

**Screenshots**

—

**Computing environment (please complete the following information):**
 - OS: macOS Big Sur
 - Browser: any browser
 - Browser Version —
 - Ruby Version: 2.6.6

**Additional context**

—

Updating this issue to cover adding a mention for how to glob image files in both esbuild and Webpack frontend bundler configs.

Speaking of the frontend developer, this can also be used with videos too? Not just images.

Thanks for the quick fix @matiaskorhonen, looks like that should work for me as well. I'd always had issues using the frontend bundler with images, only got it work correctly with CSS/SCSS.

Yeah, I don't see why not. You'd just need to add `mp4` or `webm` or whatever to the glob pattern…

I used the same setup for both images and data files on the @euruko 2022 conference site: URL

I seem to have issues with this as I work wit esBuild better than Webpack.

@jaredcwhite id say a copy plugin that writes to a manifest is better. The problem with the manifests that are generated by doing import statements is that the strings get inlined into the final bundle, so you would need to make a separate assets entrypoint that you never use.

Also, ESBuilds asset detection is spotty at best since the asset -> hashed asset requires some workarounds.

 URL

[Feature Request]: For entrypoints handled by "file" loader, add the hashed file to the metafile output
The title is hard to understand, but basically im looking for a map back to the original file that creates an asset chunk.

Example:

js
// index.js
import("academic-cap.svg")

in my ESBuild metafile:

js
for (const [hashedPath, details] of Object.entries(result.metafile.outputs)) {
 console.log(hashedPath, details)

 // public/esbuild-builds/chunks/academic-cap-JO6NDN2A.js
 // {
 // imports: [],
 // exports: [ 'default' ],
 // entryPoint: 'app/javascript/assets/academic-cap.svg',
 // inputs: { 'app/javascript/assets/academic-cap.svg': { bytesInOutput: 56 } },
 // bytes: 204
 // }
 }

If you notice theres no hashed `"public/esbuilds-builds/assets/academic-cap-[hash].svg"`, it only has the ".js" entrypoint. I'm curious if it would be possible to support the use-case of keeping a referenced to the hashed-file with the original extension and not the generated JS file.


Question: Decimal * Uint128 treats Uint128 as a raw Decimal value?

Checked math only makes sense if the contract actually handles the error cases. If they just `?` them to top level, you can also just panic. In both cases the contract execution is terminated, the transaction is reverted and an error message is logged. When you multiply with constants like 1/3 you know the operation can never fail. So I'd implement both for consistency with other math operations.

Decimal * Uint128 treats Uint128 as a raw Decimal value
Seems to be a "feature" so raw `Uint128` values can be treated as decimals.

rust
// NOTE: using a custom Decimal fork here since the constructor is normally private
 unimplemented!(
 "{} {} {} {}",
 gas_cost * Decimal::one(), // decimal * decimal
 gas_cost * Uint128::from(1u128), // decimal * u128
 gas_cost * Decimal::from_ratio(Uint128::from(1u128), 1u128), // decimal * decimal(u128 / 1)
 gas_cost * Decimal(Uint128::from(1u128)), // decimal * decimal(u128)
 );
//1 'not implemented: 0.00000000000338 0 0.00000000000338 0'

This is very surprising, since the `Div` implementation doesn't follow this behavior, and `Mul` doesn't this either.

Similar to #1155 I propose the `Decimal()` constructor to be made public. The `Mul` implementation should be fixed, then the old behavior can be more explicit via `decimal_val * Decimal(uint_val)`.

Thanks for brining this up. I'm really unsure what the desired output type is.

There is also no generic and safe conversion
* **Decimal -> Uint128:** because information after the decimal point get lost
* **Uint128 -> Decimal:** because the range of Decimal is smaller since 18 digits are reserved for the data behind the decimal point

This makes it hard to say: Use this output type that can easily be converted.

Looking at URL it feels like this should not be implemented at all.

This should be implemented. I'd expect `decimal * uint128` to do a simple multiplication. Equivalent to `decimal(decimal.0 * uint128)` or `decimal * decimal::from_ratio(uint128, 1)` but more gas efficient than constructing a `int/1` fraction then multiplying two fractions together.

i.e. I have `$0.05` dollars (internally, stored as `5` cents, the lowest possible fraction, with a `DECIMALS` value of 2). I double my money (`2 * 5`) => `$0.10`

Basically, there are two ways we could want.

200 * 50% = 100 (multiply my number by some decimal) - this is implemented as majority case
25% * 3 = 75% (multiply my percentage by a number to get a bigger percentage) - this is not typical, but should be possible.

Maybe this is included in #1186 ?

But if we have `Decimal * Decimal -> Decimal` that should be plenty. And you can convert your integer to a decimal with no loss before multiplying. Making it possible, even if not super simple syntax.

The mixed type arithmetic Decimal * Uint128 should be removed when we get the chance. Created a ticket for 2.0: URL

Remove mixed type arithmetic Decimal * Uint128
There is no natural return type for Decimal * Uint128. It could be Decimal, Uint128 or even Uint512. See URL for a more elaborate description of the problem.

The choice of Decimal * Uint128 = Uint128 has no reason other than its first use case. It has often confused users who expect Decimal * Uint128 = Decimal.

The better solution for the integer math is something like:

rust
let a: Uint128;
let b: Decimal;

let c = a.multiply_ratio(b.numerator(), b.denominator());

The better solution for the decimal math is something like:

rust
let a: Uint128;
let b: Decimal;

let c = Decimal::try_into(a)? * b;

This is not a "better solution" in my mind. Very cumbersome to use and not adding clarity to devs.

If you want to make `let c = a.multiply_ratio(b);` that starts getting usable (if verbose).
People should not be forced to deconstruct Decimal to be able to do basic math

Yeah, this was not meant as the final API. It should just showcase how `Decimal * Uint128 = Uint128` and `Decimal * Uint128 = Decimal` can be done and the implementation for both operation is there. It can be implemented with something like

rust
fn multiply_by(self, rhs: Into ) -> Uint128

instead to keep the caller code concise.

Along these lines, it would be very useful to have some ceil and floor multiplication for Uints and Decimals. Something like this:

rust
fn mul_ceiled(self, rhs: impl Fraction) -> Self;
fn mul_floored(self, rhs: impl Fraction) -> Self;

This would allow for a lot of flexibility when dealing with rounding errors etc.

@uint is 👆 something a macro implementation could help us with? I think we need those 4 functions implemented for Uint64, Uint128 and Uint256:

rust
fn mul_ceiled(self, rhs: impl Fraction) -> Self;
fn mul_floored(self, rhs: impl Fraction) -> Self;
fn checked_mul_ceiled(self, rhs: impl Fraction) -> Self;
fn checked_mul_floored(self, rhs: impl Fraction) -> Self;

With those in place, we can confidentally remove the mixed type operator implementation in 2.0.

Probably! Like an `impl_arithmetic!(u64)` macro called in every `Uint*` module?

Yeah, or better specifically for the new additions `impl_mul_arithmetic!(u64)` such that we get them implemented consistently. We discussed deduplicating the Uint* implementations before and I think this is a good opportunity to try this approach. But I'm not that confident writing macros myself due to lack of experience.

Note that Uint512 should not get them since it does not have a `full_mul` implementation. If we add that we also need Uint1024 and the series would go on indefinitly.

Isn't `Uint128::checked_multiply_ratio` already a ceil multiplication?

@webmaster128 I took a shot at it yesterday. I think it's not as simple as reusing the logic in `checked_multiply_ratio`. E.g. if you have

rs
let left: Uint64;
let right: Decimal; // or even Decimal256
let result = left.checked_mul_floored(right).unwrap();

I suspect the multiplication of `left * right.numerator()` might overflow quite easily, and `full_mul` doesn't help.

Right, this is something I vaguely imagined to become a problem.

So what we do currently support is basically

Uint64 * Fraction
Uint128 * Fraction
Uint256 * Fraction

with (more or less) Decimal being a `Fraction ` and Decimal256 being a `Fraction `. Also by widening we can easily support

Uint128 * Fraction
Uint256 * Fraction
Uint256 * Fraction

So the number if bits in the integer needs to be >= the number of bits of the decimal. We are lacking an easy way to get

Uint64 * Fraction (i.e. Uint64 * Decimal)
Uint64 * Fraction (i.e. Uint64 * Decimal256)
Uint128 * Fraction (i.e. Uint128 * Decimal256)

However, if we do not do the multiplication size based on the integer size (current `self.full_mul` behaviour) but based on the larger of the two, we can get this to work. A simple implementation just takes 256*256=512 bit multiplication to cover all the cases at once.

Hey folks, tried my hand at this here: URL for Uint128.

A few things:
- It looks like the primary use case in this issue with the Uint# * Decimal use case. For our contracts, we are trying to support a Uint128 * Fraction(Uint128, Uint128) use case ( URL This means, we need a struct that implements "Fraction". I tried implementing a generic struct for this (renaming the interface to Fractional), but the `inv()` method is problematic. Would like some feedback on this approach, possibly another direction is better.

- This does not support multiplying higher bit numbers. Any suggestions on how to handle this? Do you suggest casting everything to a Uint512?

- I made the unchecked versions simply the unwrapped of the checked. I wonder if these are necessary at all though and checked should be the default with the consumers unwrapping themselves.

Just pushed an update to that PR that adds a macro. If it's acceptable to cast to Uint512 for all int sizes, we simply need to wrap all of the Uint structs in `fraction_math!(Uint128);`. Think this is what folks were suggesting earlier in the thread.


Question: GraphServiceClient.Users.GetAsync() returns nothing or non-helpful value for Schema Extension Data?

Thanks for raising this @drewid

Any chance you can confirm the method you are verifying the properties returned from `result` object? If you are serializing the object the object, are you using this?

cs
var jsonString = var jsonString = KiotaJsonSerializer.SerializeAsString(result);

GraphServiceClient.Users.GetAsync() returns nothing or non-helpful value for Schema Extension Data
**Describe the bug**
Very related to Issue #2305 when running the standard query and "not" explicitly including the schema extension data, nothing is returned for the schema extension data. One step more problematic than #2305 is when the schema extension **_is_** explicitly requested it returns non-helpful data instead of the actual data stored and that the documentation is referring to.

When doing a regular/default query **GraphServiceClient.Users.GetAsync()** it is returning no extension data field/value/reference of any kind and only non-helpful data when the Schema Extension item is explicitly supplied by **QueryParameter.Select**

image

**To Reproduce**
Steps to reproduce the behavior:

**IDE Setup**
- Visual Studio 17.8.3
- .NET 8.0 C# command line project

**Expected behavior**
**Issue 1**
If data exists in the specified Schema Extension for the user, the data should be returned by the default query.
Should not require explicitly returning that field to display the data.

**Issue 2**
If data exists in the specified Schema Extension for the user, the actual field values should be returned when explicitly requesting the schema extension data using QueryParameter.Select

**Steps**
- Created a GraphClient _graphClient (success)
- Added values to a user for the Schema Extension (success)

- Run GraphClient.Users[UserId].GetAsync() - returns nothing for schema extension (_failure_)

- Run GraphClient.Users[UserId].GetAsync() with schema extension explicitly supplied in QueryParameters.Select (_success_ but also _failure_)

image

**Code**
The following code returns no field/value/reference for an existing schema extension data when running the basic user query:

`_graphClient.Users[UserId].GetAsync();`

The following code returns a non-helpful, non-informational value for the schema extension data when adding the explicitly select parameter (_but has the bug where every field is returned_)

`requestConfiguration.QueryParameters.Select = new string[] { "id", "displayName", "domain_schemaextensionname" };`

image

Thanks for raising this @drewid

The API will only return a limited number of properties by default and thus the $select parameter is required to fetch those properties.
With regards to displaying the results, the models used by the SDK should be serialized as pointed at here due to the custom serialization.

Any chance you can confirm this resolves it?

I will look into things further but if more work/serialization is needed then core graph documentation and examples needs to be seriously updated as it is very wrong.

Is strange that regular serialization does not work and only works with KiotaJsonSerializer. This is not documented anywhere obvious.

Thanks for confirming @drewid . Since this is related to the discussion in URL We will close this and follow this up there

GraphServiceClient.Users.GetAsync() is not respecting QueryParameters.Select
**Describe the bug**
No matter what **QueryParameters.Select** fields are entered, the Users.GetAsync() and Users["user"].GetAsyc() query returns all the fields for a user instead of the subset. My code is close to identical to MS standard examples.

**To Reproduce**
Steps to reproduce the behavior:

**IDE Setup**
Visual Studio 17.8.3
.NET 8.0 C# command line project

**Steps**
1. Created a GraphClient _graphClient (_success_)
2. Added values to a user for ExtensionAttribute1 and ExtensionAttribute2 (_success_)
3. Added Directory Extension (_success_)

4a. Return the fields Id, DisplayName, OnPremisesExtensionAttributes (_**failure**_)
4b. Return the fields Id, DisplayName, extension_clientid_extensionname (_**failure**_)
4c. Return just the fields Id and DisplayName (_**failure**_)

**Problem**
No matter what is put in the values for **QueryParameters.Select** all the fields are returned for the user instead of the subset specified in the Select array.

I have tried changing the case of the fields (Camel Case vs Capitalized) to no luck.

**Expected behavior**
Only the fields that were specified during the select should be returned.

**Code**
If applicable, add screenshots to help explain your problem.

image

`
var result = await _graphClient.Users["userid"].GetAsync((requestConfiguration) =>
{
 requestConfiguration.QueryParameters.Select = new[] { "Id", "DisplayName", "OnPremisesExtensionAttributes" };
});
return result;
`

The response returns the full list of user fields. Example snippet:

image

I printed out the serialized results using the following:
` Console.WriteLine(JsonConvert.SerializeObject(result));`

I would not expect to see all of the fields having been returned.

When I used the following you mentioned, it is empty
`var jsonString = KiotaJsonSerializer.SerializeAsString(result);`

Thanks for getting back @drewid
Any chance this shows the values returned? Due to the backingStore, the properties may not show as the serializer will pick out only changed values after the response.

cs
result.BackingStore.InitializationCompleted = false;
var jsonString = KiotaJsonSerializer.SerializeAsString(result);

A get should never just return changed values and that is not implied anywhere in the docs.

I will dig further but this is a royal mess.

- A default query should return all values.
- A select should return just those mentioned and not the slew being returned

@andrueastman Thank you for the magical code snippet regarding the BackingStore - I have no idea why that magically makes things work in making the serialization honor the select list (or at least properly trim the fields).

What is misleading though is that **the query still returns "every" field**, including the selected fields requested. The KiotaJsonSerializer is being applied after the fact and I can see the query is returning all the fields when doing a basic serialization of the result. This is remarkably misleading and incredibly wasteful. What has been implemented is not a select parameter, it's actually a "make sure to include these fields" parameter.

The information related to KiotaJsonSerializer and v5 breaking change regarding BackingStore needs a "lot" more information publicly. Is not listed anywhere in the normal docs and examples that I can see.

In essence, the standard described ways of doing QueryParameters.Select do not work without it (at least narrowing it down). It does include the fields selected, but also includes every other field.

Thanks for the feedback here @drewid

Just to clarify, what is indeed happening is that API is indeed only returns the fields selected(This can be confirmed using the Graph Explorer ).
However, the model properties will by default read as null by a serializer if they have not been set due to the proxy mechanism used by the backing store. The `KiotaJsonSerializer` is built for this and can be used to get back the Json received or to be sent out by a given execution call.

This indeed should be better documented, and we've created URL to update the v5 documentation here.

Better documentation will be great because it is 100% not intuitive and not at all clear.

Thanks for the feedback here @drewid. We'll close this for now and track this via URL


Question: [CRITICAL ISSUE] Standardization of formats, primarily NBT?

Closing this, as it's solved in the last few commits. NBT Editor allows loading of .nbt files, which store Infinity/NaN properly. Problem solved :D

[CRITICAL ISSUE] Standardization of formats, primarily NBT
Due to the recent shit with Meteor Client (see URL I think it's time to form some sort of standard for the Archives, specifically the NBT.

Currently all the NBT is sourced from Meteor Client's `.nbt copy`, and easiest to plug-and-play with Meteor Client's `.nbt set`, however the latter command is currently broken, and has led me to wonder if that's the right path anyway. Should the NBT be kept as is, or should it be updated to be plug-and-play with IBE Editor instead? This would involve updating some 81 kits, not counting open [New Kit] issues, but could make it easier. At the same time, the NBT can currently be used in IBE Editor, you just need to add or alter the `tag:{}` portion of the SNBT within IBE Editor.

If we update the 81 kits, I suggest simply adding `tag:` at the beginning of them all, and when users copy-paste they can choose to include or exclude that part.

In relation, the Using NBT guide should be updated accordingly with this.

_After a bit more research, it's incredibly easy to paste a kit into IBE, provided it has `,tag:` appended._

### Breakdown:
- By default, items do not have a `tag:{}` field. For example:
 - An orange shulker box has the following: `{Count:1,id:"minecraft:orange_shulker_box"}`
 - If our NBT data starts with `,tag:`, followed by the actual NBT itself, then users can simply click between the closing quote of the item name and the closing curly bracket, and paste in the data.

- Relatively few items have naturally existing tag fields, with the majority falling in the Combat, Enchantment and Potion/Effects categories.

- While we do have some standalone items, including some that have natural tags, the majority of our archive is storage-based, thus prefixing all NBT with `,tag:` would render the mass majority of the archive instantly usable regardless of interface (IBE, Meteor, Wurst, etc).

This should (generally speaking), solve the issue of multi-interface compatibility. Appending everything with `,tag:` makes all the kits usable in any interface, and while some items won't be usable if you straight up copy-paste the entire file, users can simply exclude `,tag:` in their selection when copying.

There are however other standards that need to be set, including the following:
- Kits/Items without Authors
- Kits/Items without Names
- Kits/Items with NSFW or 18+ content including links, text, images, references.
- Books
- Kits/Items with the ability to significantly harm servers, primarily the Total Freedom network.

## **_Please give feedback either in this thread or on Discord for the NBT Standard as well as ideas for the other standards as I can not add more kits to the archives until (at the very least) the NBT Standard is set._**

Well despite the absolute lack of feedback from anybody, I'm approving the `,tag:` standard, and will elaborate on other standards in the below comments.

# Kits/Items without Authors

This has become an increasingly problematic issue, as many of the kits saved in Antis plot lack credit of any sort.
Thus far, I've been listing unknown authors as 'Pending', but I feel this is both unclear, and a temporary measure.

### Proposed Solutions
- Switch 'Pending' to 'Unknown', 'Unknown Authors', or something clearer.
- Add a notice of some sort requesting information on kits with Unknown Authors.

In my opinion, there is no good solution to this issue, as I seriously doubt we will ever identify the authors of more than a handful of kits. However, we can certainly do a better job than at current with labeling these kits appropriately.

# Kits/Items without Names

Similar to "Kits/Items without Authors", this is a problematic issue with many kits in Antis plot, but one we can do a lot more to rectify.

### Proposed Solutions

- Names should be suggested based off the contents of the kit
- Chosen names should be agreed upon by 3 people (Kaddicus, Parsa and Anti at the very least)
- When added to the site, the name should have an asterisk (*) next to it to indicate the name is not the original
- A note should be provided in the kits description or above the list of kits stating that kits with an asterisk had no name, and the name given was chosen by the Archivists

This would allow a number of interesting kits currently sitting in Antis plot to be added to the archives.

# Kits/Items with NSFW or 18+ content including links, text, images, references.

Coincidentally this is an issue being currently discussed on the server: [[POLL] The Matter of NSFW Content]( URL

My views have already been expressed in that thread, but that's for the server as a whole.

My view of NSFW and the Archives is that anything goes, _provided it has suitable warning_.

### Proposed Solutions
- Add a neon red [NSFW] tag on kits containing NSFW
- Separate NSFW kits into a separate page, or add a checkbox to show/hide NSFW kits (hide by default)

Ironically, the solution to Issue #63 (NaN/Inf Attributes not saving in SNBT) invalidates this entire issue. Unsure if I'll reverse the NBT format decision, given how fucking long it took me to implement, but we'll see. Closing this issue by tomorrow when I decide.

# Final Changes

Ok, long-ass post incoming. Thank fuck GitHub has a fairly massive char limit, cus I got a lot to say.

## Summary

This entire issue started when Meteor Client's `.nbt set` command broke (fun fact, they still haven't fixed it), prompting me to switch everything over to IBE Editor. However, not even a few days later, I started investigating Issue #63, and found that any form of SNBT would not save certain attributes, and in the case of #86 SNBT would often just shit itself when you tried working with books. Aside from being problems in and of themselves, they also meant that this issue was partially invalidated since SNBT would no longer be the primary format of the archives.

## Changes

The initial goal of this Issue was to update all of the archived Kits and Items SNBT to include the `,tag:` used by IBE Editor.

### SNBT

I actually did carry through with that update, and all the SNBT in the Kits section of the archives contains it.

However, I then went on to upgrade every NBT page (as part of #63's swap to .nbt files), and in doing so made the SNBT copy-able with the click of a button.... except it now contains `,tag:` which means you can't use it with Meteor Client's `.nbt paste` or NBT Editor's SNBT menu unless you manually select and copy the SNBT and exclude the `,tag:` bit.

### Kits without Authors/Names

I still intend to implement some of these ideas, but never got around to it because of the SNBT/InfNan business.

## So... what now?

Well, I've made a royal mess of things haven't I.

To start off with, I'll reverse my initial decision, and remove `,tag:` from all of the SNBT. This means I will also need to extensively edit the just-finished UsingNBT page, but so be it. I'd rather have the SNBT be easily copy-able for the few people who use it than be a swiss army knife of Editor Support.

For the Kits without Authors/Names, I'm going to implement the following:

- Switch 'Pending' to 'Unknown Author'
*I feel this is a much better term than Pending, especially given that we may never know the Authors of some kits*

- Have a group of us (Anti/Parsa/Myself, possibly the greater extent of PCI as well) choose and confirm names for unnamed Kits
*It will take more thought, and some organization, but I feel this is the best way to handle this. There are plenty of amazing kits in Antis plot alone being excluded from the Archives because they have no name, and I'd rather they have an unofficial name, than no name at all.*

There is also an unposted comment I typed up pertaining to NSFW in Kits and Items. I had initially planned to lay down rules regarding that, but have since rethought the situation and decided that Kits and Items should not be excluded from the archives on the basis that they contain Mature or Adult content or themes.

## Going Forward

While the initial reasons for this Issue are either invalid or semi-valid, there are still things I want to see standardized. A good example would be Kit Submission.

The current setup we have is:
- \> Users make a Kit/Item/Book
- \> They open an Issue on this Repository, using the applicable form
- \> Fill in the data and Submit
- \> Eventually I get around to adding it.

Well, the major issue with that now is, of course, .nbt files. Even if users wanted to submit .nbt files, GitHub doesn't support uploading them in Issues/Comments (because that would just be too convenient now wouldn't it?).

I've no idea how users are supposed to submit kits now, but I may try some method involving a submission box on the site, and emails.

For now, the GitHub Issues method for submitting kits is going to have to be shut down, and kits will just be submitted to me or someone I authorize to receive them in-game. *Oh how I wish we could use /mail for Kits...*

AnYway, that's going to be my closing comment for this. I have more to say, but it's an unformed incoherent mess when I try and type it, and I'm getting tired of typing, so yeah.

***Issue Closed***

Infinity/NaN
Minecraft's JSON will just consider those strings. Infinity strings can be replaced with a valid (although pretty big) number. As for NaN though, well that is not gonna work. Maybe a different approach is needed

@unrealiceyou both are rather uncommon in the wider spread of kits (or so I believe), so I'm not super concerned. I can probably just preface the text file containing the NBT with a note about the Infinity/NaN when needed, including how to properly import it.

Can confirm after testing that both `Infinity` and `NaN` just gets converted to `0.0` when added to an item as SNBT, and has to be manually changed.

Ultimately it seems Infinity/NaN items can't be stored as SNBT at all without removing the Infinity/NaN.

Unless there's another way to save these items beyond SNBT.... they can't be saved.

Can confirm, Hotbars work.

Any method involving SNBT will _**not**_ work, as SNBT causes NaN and Infinity to be reset to 0.0 when applied to an item.

My theory is when the SNBT gets converted to normal NBT, it can't map "NaN" or "Infinity", so resets it to default (0.0)


Question: DjVu documents (and potentially other cases) cause chafa to consume a lot cpu and memory?

That would be nice to have, indeed. Since it's so little code, it would be easy to audit and fuzz test too.

DjVu documents (and potentially other cases) cause chafa to consume a lot cpu and memory
**Summary**:
Since chafa seems to try to display as many file formats as possible, sometimes it might just be inappropriate and should be prevented or optimized.

**Example**:
One such case is DjVu files, many-page djvu documents to be accurate.

In my exact case, I have set in my lf preview script to use chafa to preview any file of image/* type. Unfortunately, DjVu files are "image/vnd.djvu" type. However, I think the argument applies to more general use cases as well.

When I use chafa to (pre)view a several-hundred-page djvu document, it consumes all the available memory and the system just hangs.

**Discussion**:
Maybe chafa should at least check for potential memory consumption.

Maybe also optimize the preview, such as only preview the first page, which I believe will never cost many GBs of memory. This possibly applies to PDFs as well, which currently is very slow (a few seconds) to display.

this has happened to me as well, when previewing a large pdf using lf, my window manager just froze

Hi, thanks for reporting this. This is happening because Chafa delegates to ImageMagick when there's no internal loader to handle the file format. ImageMagick gets very enthusiastic and sometimes trips on its own shoelaces.

The problem is fixed in master and will go away in the next release -- see #157.

In the meantime, you should be able to curb problematic formats in ImageMagick's configuration. On my distro, this lives in `/etc/ImageMagick-7/`. The best way is to list it in the security policy file:

1. Find the format identifier by running `magick -list format`.
2. Add it to the policy file (`policy.xml`), e.g:

sh

The default configuration varies between Linux distros; mine already has PDF on the do-not-load list, and it looks like DjVu support is not built in at all.

I just released version 1.14.0, where ImageMagick is no longer an option. That should fix the issue (but it will no longer produce previews for DjVu and PDF; maybe one day it will be possible to do so safely).

Remove ImageMagick support
ImageMagick is a complex dependency that's hard to support. While it's an extremely versatile library, it's also got a kitchen sink approach that includes spawning external processes, causing inadvertent resource exhaustion and security issues. It would be nice to remove this dependency to cut down on risk/maintenance/packaging footprint.

That would leave the `chafa` CLI tool with support for the following formats: GIF, JPEG, PNG, SVG, TIFF, WebP, XWD. We may want to add support for AVIF as mentioned in #98 first. Potentially other important image formats too (suggestions welcome).

We'd lose support for formats like PDF (which many distributions disable anyway) and all video transcoding via IM (ew!). Users who want to keep using these can pipe to stdin like in the following examples:

bash
ffmpeg -i URL -f gif - | chafa -d inf

bash
convert URL gif:- | chafa -t 1 --bg white

...and the performance will be better too.

QOI might be worth a look. It is not widespread as exchange format, but quite popular as internal image format in many apps. It has okish compression and is really fast for its small code footprint.