Question: Put more requests for help & advice on requests into public?

We could have an admin page of "allowed tags" that users can suggest or add (like our body performance tags or `pit`-related tags). That would reduce the impact of users adding synonyms or tags that generate notes.

Put more requests for help & advice on requests into public
We could have a public "get help on this request" link somewhere on request pages, which goes through to the user support inbox but also gets posted on the request thread (perhaps behind a tabbed interface).

We could then allow admins to answer the help request in public, with the aim of being able to point people towards existing answers rather than using canned responses to reply privately.

Possibly, on viewing the help request, admins could also mark it safe for public comment (rather than admin-only reply) to enable other users to contribute. We could have a feed of these help requests similar to URL

Already ideally admins (and others) will provide advice in public via an annotation.

Exposing per-request tags URL would enable the use of a tag to flag a request for help. Ideally we'd have a button to add the tag, and a page listing requests on which help has been sought.

Related:

Consider presenting the "next steps" / refusal advice publicly #6286

Previous discussion on offering help in public via a mailing list, an option it was decided not to pursue:

 URL

Often users who write to the support team seeking advice have not classified their response (or they've classified it as needing the attention of the team) whereas if they had classified it eg. rejected or awaiting a response the system would have automatically offered advice.

Doing more to encourage classification could prevent users seeking human help:

 URL

This issue has been automatically closed due to a lack of discussion or resolution for over 12 months.
Should we decide to revisit this issue in the future, it can be reopened.

Expose per-request tags publicly and enable searching and browsing using such tags
This is a request that has been made by a user of WhatDoTheyKnow.

The request suggests allowing anyone to assign tags to a request - as an option under the "Things to do with this request Anyone" heading under a request's correspondence.

Their ideas in relation to search include showing search results collected by tag within a particular body.

By not having this feature we could be missing out on a way to highlight requests of interest.

A "tribunal" tag could be used to mark requests which have been appealed to the Information Rights Tribunal for example.

 URL

Tags _are_ indexed, so that search _nearly_ works.

 URL is what you're after.

Obviously the discovery of this is poor, but its at least usable for promoting on Twitter etc.

Noting a suggestion raised during the mySociety retreat which was something along the lines of each user could be allocated a certain number of tags which they could allocate to requests.

The idea was as I understand it if tags were to be used to flag up requests of particular value / interest, they could be rationed, so any one user would have to allocate them judiciously.

I think this might be getting away from tagging and more towards an "upvoting" system.

Not crediting the idea to anyone as I might not have noted in the way they were envisaging it working.

See URL for a campaign encouraging users to tag requests via a hashtag in the request subject. The tag is then used to make it easier to avoid duplicate requests to the same body.

Presumably if there had been a public tagging feature they'd have used that.

A WhatDoTheyKnow.com user has suggested a database field for ICO decision reference numbers, enabling linking of requests to decision notices.

This could be achieved via tagging requests, with a tag format of eg.

`ICO_REF:XXXXXXX`

key:value tags are used extensively in the WhatDoTheyKnow body database, so using them when tagging requests too feels reasonable.

Consider legal and other risks of publicising existing tags where they might to draw attention, or be deemed to draw attention, to request threads on which we've hidden material.

We could start by exposing, and perhaps allowing the public to submit, tags relating to poor public body performance:

 URL

There is a potential moderation issue though.

If we did this we'd have to be a little careful about how it interacts with the special behaviour of some tags and showing notes based on a request's tags (#6912).

Let's say we have a Note that applies to requests tagged `foo`. This would likely be deducible from the exposed tags and the note content. Users could then add `foo` to other requests and the Note would appear, whether relevant or not. Do we want e.g. #7106 to be user-controllable?

We could automatically generate a moderation notification in this scenario based on the fact that we'd know an added tag relates to a Note or special behaviour, but that creates admin work. Would we then also want to tweak the behaviour of #6912 to list out content that an in-creation Note would get added to in the case that the tag `foo` unexpectedly exists on some requests?

We could use tagging to help capture PIT extensions, which could help to differentiate between requests that are illegally late, and ones where the deadline had been legally extended. This could even feed into the Pro dashboard in some way.

As do NARA over in the United States - their "Citizen Archivist" programme has volunteers doing a similar task, and makes use of tagging. They even have "missions" which are a targetted way of prioritising certain activities. Maybe there is merit in us looking at this?

 URL
 URL


Question: .so file not found?

`extension_dir` is the best place for `ffi_lib` feature. We will document it at URL

I did investigate all affected gems in rubygems.org and fixed them that released in 2022. see URL

I'll close this issue because there is no action about this in RubyGems 3.4.x.

.so file not found
Getting this error after installation in Docker container

Could not open library '/usr/local/bundle/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so': /usr/local/bundle/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so: cannot open shared object file: No such file or directory (LoadError)

The `.so` file is present in `/usr/local/bundle/gems/cld3-3.5.0/lib/` but not `/usr/local/bundle/gems/cld3-3.5.0/ext/cld3/libcld3.so`

Reproduction: URL
 URL

tl; dr: cld3-ruby won't support the latest RubyGems until RubyGems developers figure out a reliable way to get the path to the binary. I recommend to stay away of the latest RubyGems and Ruby, which includes the lastest RubyGems for now.

I authored commit 0ff7bb57a052fbdc346f28c1e1e9b097bfc606b6 but I decided *not* to merge this change. The problem of this change is that the binary won't be installed to the `lib` directory on Fedora and on Arch Linux (and potentially on other distributions as well). For details, see:
- URL
- URL

Referring to the `lib` directory can also be problematic in the future as the binary in the directory is stated to be removed anyway: URL

Unfortunately the reality is that RubyGems introduced a disruptive change without realizing its nature and there is no good workaround. I hope the situation gets better soon; only a few days have passed since Ruby 3.2.0 release anyway so let's wait a bit longer.

Is there an active/open discussion against ruby gems somewhere? I see that URL was linked, but is closed now.

I don't think so. There is a PR to document an API useful to fix the issue but it's kind of stale and does not provide an exact procedure to resolve the path: URL

@akihikodaki Please consider releasing a version compatible with ruby 3.2 even if it is not optimal for all use cases. You can release it as prerelease gem so that it has to be opted in explicitly.

Cleaning build artifact broke some of gems
### Describe the problem as clearly as you can

Related with URL and URL

I found the some of issues related object cleanup.

* URL
* URL

`ffi` may uses build artifact under the `ext` directory, not installation directory. Because they couldn't require gem library via `ffi`.

I didn't investigate `ffi` side yet.

### Post steps to reproduce the problem

bundle init; bundle add blurhash; bundle exec ruby -rblurhash -e 'puts Blurhash::VERSION'

`ffi` try load under the `ext` directory like

/Users/hsbt/.local/share/gem/gems/blurhash-0.1.6/lib/../ext/blurhash/encode.bundle

(`$HOME/.local/share/gem` is my GEM_HOME environment)

* Build files by RubyGems 3.4.1

$ fd .
CHANGELOG.md
CODE_OF_CONDUCT.md
Gemfile
Gemfile.lock
LICENSE.txt
README.md
Rakefile
bin/
bin/console
bin/setup
blurhash.gemspec
ext/
ext/blurhash/
ext/blurhash/Makefile
ext/blurhash/encode.c
ext/blurhash/encode.h
ext/blurhash/extconf.rb
lib/
lib/blurhash/
lib/blurhash/version.rb
lib/blurhash.rb
lib/encode.bundle

* Build files with RubyGems 3.3.26

$ fd .
CHANGELOG.md
CODE_OF_CONDUCT.md
Gemfile
Gemfile.lock
LICENSE.txt
README.md
Rakefile
bin/
bin/console
bin/setup
blurhash.gemspec
ext/
ext/blurhash/
ext/blurhash/Makefile
ext/blurhash/encode.bundle
ext/blurhash/encode.c
ext/blurhash/encode.h
ext/blurhash/encode.o
ext/blurhash/extconf.rb
lib/
lib/blurhash/
lib/blurhash/version.rb
lib/blurhash.rb
lib/encode.bundle

### What's the next step?

I'm not sure we should revert URL yet. Because why `ffi` uses extensions under the `ext` directory. But this change may broke many of gems depends on `ffi` some of gems.

If this change is intentional, we'd want to change lines like URL

I found another issue URL without `ffi` usage.

`ox` relied on `ext/ox/ox.{so,bundle}` from their ruby code.

I was aware of this potential problem, but I wasn't able to find out any regression in few popular gems I have tested. The question now is what's wrong. Should you rely on files in original build destination removed by `make clean` folder after successful `make install`? Per my understanding doing this is a mistake in autotools world.

Basically, I agreed to clean build directory with you.

* `blurhash`: They should use under the `lib` directory, not `ext` directory.
* `ruby-argon2`: Tha find logic of `ffi-compiler` is may wrong.
* `ox`: They use `require_relative "ox.so"`, not `ox/ox`.

I'm not sure how the impact about this. I can investigate and fix them for the current issues.

I'm looking into extensions right now to revisit how they works to make it finally clear. I'm not sure that fix is handling all the use-cases yet. Currently extensions are copied into two places per following code. URL

This also broke this gem - URL and its predecessor

Ideally this change should be opt in. There are unmaintained gems that used to work but are now broken.

That's seems like similar problem with ffi fixed path.

 URL

Well, turns out @eloyesp and @simi were right and I completely missed the case of using the artifacts directly when I deemed #6133 as "mostly compatible" at URL 🤦♂ . And it does not seem that rare in retrospective since the compiled artifacts used to live relatively to the other gem files, not in some more strange location.

As per what to do, I'm not sure. For now I can move #6133 to the breaking changes section and add a note about what to do?

Also, regarding what to do, I want to note that installing to lib was supposed to be something only temporary and it was eventually planned to stop doing that. At least there's a TODO about making that change for RubyGems 4. I think the rationale being that extensions are arch-dependent, so they should be installed to an arch-dependent folder (and `lib` is not). I think the "arch-dependent" alternative would be `Gem.loaded_specs[ ].extension_dir`.

There's also this ramification that I didn't think about:

Artifacts generated at the `ext` folder (and now automatically cleaned up after installation) are the only ones with a valid `ox/ox` require path. Now that possible require path is gone. I wonder if this is also related to #6204 🤔.

Thinking about an ideal deprecation path here, it seems pretty messy to me 😭. We would need to warn when this kind of route is `require`'d. For that an idea would be to instead install a Ruby file that first warns and then delegates to the properly installed compiled artifact. But we'd want to display the warning only to gem authors, not to gem users, since gem users can't really do much about these warnings. I guess we'd need to detect whether the CWD is the root of the gem for that or something.

Alternatively, since damage is done now and a fully clean deprecation path does not exist anymore, we could add a way for users to opt-out of the new behavior as suggested above, until gems are fixed and users can move on to the new version.

Ideally we could make this setting configurable per gem, so that users can still benefit from the savings of cleaning up other artifacts, but that's probably overkill.

I don't see why it should be opt out instead of opt in. Not cleaning up artefacts doesn't change any behaviour. It just uses extra diskspace, which is same as before.

Disk space is cheap. I spent some 5 hours yesterday trying to figure this out. Look at the number of people who commented on URL I'm pretty sure they would have happily spent that extra diskspace

This was caught in our CI since we use a Dockerfile which uses the latest rubygems. Imagine someone deploying a new machine image which does that. Apps which used to work will break in production. This latest update is a **ticking time bomb**.

Please revert this change and push an update. It can be brought back as an opt in.

@j-manu I have explained the root of the problem at URL I don't see any place (I was looking into already) where anyone recommends (even outside of RubyGems) to use those intermediate folder artifacts. I'm still wondering why that is used in those listed gems.

Originally this was reported by unhappy users with 1GB+ gem directories with intermediate build artifacts of grpc for example ( URL or similar trouble mentioned by sassc gem if I remember well causing troubles with other usecases (tiny containers, ...).

@simi Ok. I didn't know there were gems with such high disk usage. It makes sense for those gems.

But my point is that any system where an affected gem is used and rubygems are updated to latest (like in a setup script) will break.

Doesn't it make more sense to revert the change and bring it back as opt in?

I also spent over the 5 hours to fix them. Don't brag about the time spent. We are considering what's the best way and for the Ruby's ecosystem.

We're considering what's best, the fact that a change breaks something for someone does not necessarily mean that it needs a revert. Even if we all believe this change is a right move that benefits the majority of the community, we need to evaluate how common it is to depend on the old behavior, and how common this is going to be in the wild and make a decision.

Of course, this only affects people not locking their dependencies, so a quick fix for people out there is to lock their dependency on RubyGems, which is a good practice for all your external dependencies anyways.

It is hard to say, since personally I would not recommend to rely on `gem update --system` anywhere in automation for various (security, reliability, maintainability) reasons. For now feel free to replace that with `gem update --system 3.3.26` or any other version working well for you.

As mentioned before, this is not easy to make opt-in/opt-out and after release it is even much harder. Also as mentioned before, I still do see using intermediate directory artifacts as a undocumented/not recommended way of handling gem extensions. If you have any insight, feel free to share to make it easier to decide.

I'm not bragging about time spent. I'm comparing the time spent by various people including you to the advantage of this change - diskspace saved.

Please understand that I'm not trying to blame you or any of the maintainers. The change was made in good faith and I appreciate the effort all maintainers put in

We all have the same goals. As you said the health of Ruby's ecosystem.

This is a breaking change. The gems affected may not have been following best practices but they worked before this change and now don't.

The best practices I know for deployment involves locking the Gemfile and locking the Ruby. I have never locked the rubygems.

I have already solved the problem for myself. I am only asking for a revert and opt in because I think people might find out about this by their apps breaking.

I understand the pragmatic side of this, and we generally try to follow it. Just want to note that according to that definition, _any_ change is a breaking change. Because anybody could be depending on anything. A breaking change is a change that breaks public, documented usages of software, like APIs or any other contracts.

I think that we are discussing many intermixed issues, and it is hard to solve them all with a single patch. So I would like to split the problem in parts:

- There is no documented folder structure for installed ruby gems so every user use a different strategy to access built extensions. To fix this we need to document a public API to access built extensions, like: `Gem.loaded_specs[ ].extension_dir`, that works. By getting the API clear we make sure that most people interact with the gems with a simple entry point. We could also "promote" that API on the `ffi` docs and examples.

- We need a consistent "extensions" path, accessible by that is correctly added to `LOAD_PATH`, as currently that folder is no consistent for git sourced gems installed with `bundler` (see here).

- On the other hand, we need a patch for the current version to make it easy to workaround current issues with unmaintained gems. We can add the `--dirty` flag that disable the `clean` step altogether.

May be I'm missing some problem, but hope that the idea of _divide and conquer_ helps here.

These gems were not depending upon a private API but a behaviour. Rubygems did not delete build artifacts before but now it does. I'm struggling to come up with an apt analogy but it is like automatically running `make clean` after `make install`.

I found a couple of other gems which are probably affected by the same problem
* URL
* URL

@j-manu Please report the same problems after confirming and investigating them.

`modelfox` case happened at Dec 21, 2022. That date is before the date of RubyGems release.

This is a bit of a philosophical discussion, but again, every change changes some behavior from something to something else. So every change is a breaking change if you depend on the old behavior. To give a stupid example, even we released RubyGems 2.4.2 to be exactly the same as RubyGems 2.4.1, except for the version number, that would be a breaking change for a gem `foo` implemented as `raise if Gem.rubygems_version > Gem::Version.new("2.4.1")` 🤣.

The way I see it, not deleting build artifacts was an internal change unless we explicitly documented that the build artifacts are meant to be used (which we don't, as far as I know). Our understanding is installation artifacts installed by `make install` is what is meant to be used, not anything else.

That said, we understand some fairly popular gems were relying on this even if not officially supported in any way, and we're willing to minimize the disruption for people, so we'll make this work again somehow, at least temporarily, until we have a less disrupting way to do this.

Breaking Change - We will agree to disagree on this 🙂

Regarding using `gem update --system` without pinning a version, here are a few examples of it being used

* Discourse
* Plots2
* Rswag
* Cloudserver
* Shuttle
* Plaid Ruby

It's a very common pattern.

So what should one do here, realistically? Just starting an unmodified Docker image of `ruby:2.3.0` will fail to install and load some gems because of this change.

Should I downgrade Rubygems inside the containers? Which version should I use? Will it work on Ruby 3.2.0 to backport to an earlier version than the shipped version?

Or is the solution to fork problematic gems and fix it there? What does the fix look like? Should the compiled files be moved into `lib` until we have a better option?

Sorry for being a bit dense here, but I find the discussion here to be pretty spread out over multiple things, and I just want to be pragmatic for the moment to get things working again.

Thank you.

@Mange ignoring your tone of voice, can you share details on what's your problem? Which gems? Have you tried to update to latest gem versions?

Hi, I'm a developer of an affected gem (cld3). First thanks for working on this issue and proposing a fix for cld3. For cld3, this issue is tracked at: URL

I'm sorry but I decided not to apply the proposed fix, which uses the `lib` directory as an alternative. One of the problems of this approach is that the copy in the `lib` directory is expected to be removed in the future, which is already noted in this issue.

But I also found another issue: apparently some distributors patch RubyGems so that it won't create the copy at the `lib` directory. The below is the patches included in Fedora and Arch Linux:
- URL
- URL

Eventually I decided not to support the latest RubyGems until a reliable way to get the binary is determined. It's been only a few days since Ruby 3.2.0 was released and I can wait a bit longer.

@akihikodaki Did you try `Gem.loaded_specs[ ].extension_dir`?

@deivid-rodriguez @simi I'm aware of the interface but I don't introduce it to my gem yet because it seems an API not documented and I haven't seen a consensus that it is the way forward though I do think it is a promising option. I'm not in hurry so I'll just wait until the interface to be used gets decided.

That's the proper way of storing compiled extensions. Anyway feel free to wait if you're not in rush. I do plan to upgrade some guides to make it clear what's recommended way of handling extensions, since we have found out (thanks to this issue) there are various practices applied and not all of them are the good ones.

Btw. IMHO in your case the best would be to not vendorize the gld3 at all, and ask client to install it into standard system paths on its own (using system package manager or by compiling and installing manually). But I understand that would add additional steps to the client side and it is not possible everytime.

@simi I think I will wait for the updated guide; it would help greatly.

cld3 is not designed in a way it can be installed as an independent library. Chromium and the Python binding just takes source files, compiles them, and link them directly. It also has a CMake build script, but it's just for compiling and doesn't have install rules (it doesn't say what header files to be copied and where the copies should be placed, for example) so it's not really useful. cld3-ruby also has its own code to convert cld3's C++ API to C so we need to build a shared object anyway.

I proposed URL to get `Gem::Specification#extension_dir` documented.

I apologize if it came off wrong. I tried to formulate myself as formal and technical as possible and just focusing on practicality, but I suppose that I didn't do it well. I'll reflect on how to make it even more cold in the future! :-)

Some old gems cannot be installed on a newer version of Ruby.
I want help them to fix it, but I'm not sure what API to use in any potential patches I want to send in.
I think I would want an "upgrade guide" or "here's how to install shared libraries in your gem" instruction. Is this documented anywhere?

---

I should add that I'm super happy for this change. I was one of the people that had real problems with `grpc` adding gigabytes of build artifacts when installing and actually migrated away from some Google services partly for that reason. I look forward to having this change implemented.

@deivid-rodriguez this conversation has prompted a question...

It makes sense that `Gem.loaded_specs[ ].extension_dir` should be the place to put these extension artifacts. However, is there a way for the gem to know what ` ` is without hard-coding it or passing it in as a parameter? Like, to be detected from context?

I was looking at the ffi-compiler code to see if there was a way to get it to output to the appropriate directory, as a strategy to get support to be automatically added to old gems that use it, like scrypt. Right now, ffi-compiler outputs to the a subdirectory of the source directory, which might be ext. At least one of the gems mentioned above has run into this issue because of artifacts created by ffi-compiler. I'm trying to figure out how ffi-compiler could detect the right output directory if it's being used to compile an ext, and again when it's being used to load the stuff it compiled earlier. But to do that, it need to know that ` ` to get the right directory.

Btw, afaik scrypt is not affected by this issue despite ffi-compiler putting the output files in ext, because it doesn't use "extconf.rb", so its generated code isn't deleted from ext. Still, it would be better to output to the right directories.

Another gotcha is that ffi-compiler isn't really maintained anymore, and the newer fork has a different name, which means that it's not usable by old gems which reference the original name. So I'm working on trying to fix the issue in a vendored ffi-compiler gem until it gets maintained again.

@BrianHawley I'm not sure if that's possible, I think the full name of the gem would need to be passed as a parameter since there's one extensions dir per gem. For example, from my Ruby installation:

/Users/deivid/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/extensions/arm64-darwin-22/3.2.0/
├── byebug-11.1.3
│   ├── byebug
│   │   └── byebug.bundle
│   ├── gem.build_complete
│   └── gem_make.out
├── commonmarker-0.23.6
│   ├── commonmarker
│   │   └── commonmarker.bundle
│   ├── gem.build_complete
│   └── gem_make.out
...

At URL @hsbt linked to `ffi-compiler` code and suggested we could do something about it there, but then the few fixes have gone directly to each affected gem so not sure if this is fixable inside `ffi-compiler`. Sorry I'm not able to help more, please share what you find.

Hi, the fast_excel gem uses ffi to compile libxslxwriter and is also impacted by the change.

There is a PR URL which removes the `clean` task. As @Mange pointed there, I also think it's not the right move, but then I don't know how to fix, because from what I read in this issue, it seems to me the path for loading libxlsxwriter is the good one: URL

I'm clearly a noob about how a gem should build extensions, install, clean, etc.. so if anyone here could tell if the related pr in fast_excel is the solution, or if the shared lib should be built at an other place then correctly loaded by ffi.
Maybe @deivid-rodriguez, @simi, @BrianHawley ?

cross referencing some other extensions:
* levenshtein-ffi has two proposed fixes
 * URL
 * URL
* murmurhash3
 * discussion issue URL

Given URL I think I can help you, @sly7-7.

You're already doing the no-code extconf.rb trick. I think that the `sitearchdir` var can be used to make the Makefile `install` target actually do something, then you can use `Gem.loaded_specs["fast_excel"].extension_dir` to get the directory for the file you pass to `ffi_lib`.

There's nothing in that PR that makes sense, except the title. I'll make a new one if you don't get to it first.

@BrianHawley Thank you for the help, I think I've finally managed to find an acceptable fix. If you have the time, could you please review URL and see if that makes sense ?

With a few days passed I decided to revisit this issue for my gem but the situation does not seem good. People are inventing different ways to fix this one problem, which may be creating potential compatibility hazards in the future. The below is candidates of fixes I have seen so far:
- Iterate `$LOAD_PATH`: URL
- Search `lib` and fall back to `ext`: URL
- Read `extension_dir` and fall back to `lib`: URL

@akihikodaki I've done it that way because reading this issue, it seemed to me this is the direction rubygems is taking (mostly seing URL

I also think `extension_dir` is the most promising one, but it is not clarified yet if we can say it is the better one than other options.

Also, it is unclear what kind of fall backs we should have. Falling back to `lib` will allow running the gem during development without installing, but there are some caveats:
- You need to copy the binary to `lib` during development.
- It does not work in the case where binary is copied to global `sitearchdir`.
- If you write the code to read `extension_dir` in a wrong way, the code may still work because of the fall back and the bug may not be noticed before someone tries to install the released gem to Fedora, which does not copy binaries to `lib`.

So we may need to consolidate the method to fall back when RubyGems is not in use, or just decide to iterate `$LOAD_PATH`; it is more complex but solves all of these problems.

My plan is to add recommendations to RubyGems guides as well. Feel free to close or leave open until that is done as well.

As a temporary **local fix**, I went to the gem dir and compile myself, using `make`.

/home/fidalgo/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/home/fidalgo/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so': /home/fidalgo/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/libcld3.so: cannot open shared object file: No such file or directory (LoadError)

So I went to `/home/fidalgo/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/cld3-3.5.0/lib/cld3/../../ext/cld3/` and just run `make` and voila, app loads again.

NOTE: This is a local workaround only.

You can also just copy the already-compiled file from the gem's lib/ directory. We ran into this on the `threatstack-agent-ruby` gem, which we can't fork and fix -.-


Question: StickyHeader scrolling issue.

Can't replicate the issue on Flutter v3.0.0 and sticky_headers 0.3.0. Please retest and if it's still a problem provide example so we can resolve it.

StickyHeader scrolling issue.
### Can we access your project?

- [X] I give permission for members of the FlutterFlow team to access and test my project for the sole purpose of investigating this issue.

### Current Behavior

I have a scrollable column, with a StickyHeader inside it. The content of the StickyHeader widget is a listview. I have datapills in the header of the StickyHeader, which filter my listview below. If one scrolls down in the list view and switches to another datapill and the filtered list items length is shorter, the header of the StickyHeader jumps to the bottom of the page and resets back only if the content is touch scrolled. I tried hiding and displaying the header again after the filter is applied, but the behavior is the same. I also tried scrollto action on the content view, that didn't seem to do anything either.

### Expected Behavior

The conent updates based on the filter change, but the sticky header sticks to the top.

### Steps to Reproduce

1. Create New Page, add navigation to the page from home page. (You may be able to do it on Home page as well instead).
2. Add new Page State variable showItems, Boolean set to true by default.
3. Select the column in the new page. Make it scrollable, select stretch for Cross Axis Alignment.
4. Add StickyHeader widget inside the column.
5. Add a Button into the header, wrap it in a row. Set the action on it to to update page state toggle the value of showItems
6. Select the last sample list item container in the content of the sticky header, make its visibility conditional equal to page state variable showItems.
7. Duplicate these a couple of times so the list items are scrollable down the page.
8. Launch the application, navigate to the test page, scroll down in the list view then push the button in the header. You will notice that the header will jump to the bottom of the page.

### Reproducible from Blank

- [x] The steps to reproduce above start from a blank project.

### Bug Report Code (Required)

ITFXi87huJdgrthE+qniLMZonmEwGVwbbZIRiO4aTTEjGJz1BIYMaffSS1pBONOkTWw2KEWKpHsE0uj3ivP9IMYBNSquUapCwatUZj3vWky/aoSTBsyaQ0QnAdpbGhG90YmsniJdCNNhdEIs7DiMBeeRSindNKH6Zwh9f6PHaOI=

### Visual documentation

<img width="402" alt="image" src=" URL

<img width="384" alt="image" src=" URL

### Environment

markdown
- FlutterFlow version: v5.0.2
- Platform: Web, iOS, Android
- Browser name and version: Chrome 129.0.6668.60
- Operating system and version affected: MacOS Sequoia 15.0, iOS 18

### Additional Information

Trying to publish my updated app with this new page in it.

Hi @fabianjozsa, I tried to used sticky header many time, however always opted at the end for a stack, reason being was that annoying jiggling effect that the header has when you scroll the list below (even when setting fixed hardcoded height and width sizes). Have you find a solution for that? Do you get that jiggling jumping like weird thing it does when scrolling?

I mean this one:

 URL

Anyone from support knows how to fix that header moving so to be "sticky" without moving and shaking?

Hello!

I attempted to replicate the issue but was unable to do so.

Here are the steps I followed:

1. Created a new blank project.
2. Set the Column on the default page as Scrollable.
3. Added a StickyHeader.
4. Added two buttons to toggle the value of an AppState variable.
5. Filled the list with multiple containers to enable scrolling.
6. Ran the project in Test Mode.
7. Scrolled to the bottom and tapped the button.

Could you confirm if there’s anything I might have missed in these steps? I’m happy to investigate further!

Looking forward to your feedback. Public Project you can clone to test and modify: URL

Please watch this video that shows the issue in action:

 URL

Didn't mean to close the issue. Please watch the video.

 URL

**Hello,**

Thank you for the video. I was able to replicate the issue on my end. However, I'm still not certain whether this is a Flutter-specific problem or something that FlutterFlow can address. I'll forward this to our engineering team for further investigation.

I just found out this is a known issue with the package ( URL I'll add this as evidence to see if the engineer team may apply a solution

I reported this issue and others related to Sticky Header, you can check it here:

 URL

Mate that issue was opened in Jan 25, we are now in October, maybe would be good to merge them all into one and push them back up top. @Alezanello what do you think?

Headers don't get stick to top after disappearing keyboard.
sticky_header_issue

Headers don't get stick to top when I try to disappear keyboard . it does not work as it should do. sticky header overlaps its own content . when I scroll little bit then it goes to its place on top as you can see in uploaded Gif.

still waiting for the solution.. because i have the same issue..

I can reproduce when entering to other page then go back. I believe it's same root cause.

Hi, I'm currently experiencing the same issue mentioned above.
Currently on Flutter v3.3.6 and sticky_headers 0.3.0+2.

Here is the full code:
import 'package:flutter/material.dart';
import 'package:sticky_headers/sticky_headers.dart';

void main() {
 runApp(const MyApp());
}

class MyApp extends StatelessWidget {
 const MyApp({super.key});

 @override
 Widget build(BuildContext context) {
 return MaterialApp(
 title: 'Flutter Demo',
 theme: ThemeData(
 primarySwatch: Colors.blue,
 ),
 home: const MyHomePage(title: 'Flutter Demo Home Page'),
 );
 }
}

class MyHomePage extends StatefulWidget {
 const MyHomePage({super.key, required this.title});
 final String title;

 @override
 State createState() => _MyHomePageState();
}

class _MyHomePageState extends State {
 final firstList = ['Hello', 'What is up?', 'You alright?'];
 final secondList = ['Bye', 'Adios'];
 final thirdList = ['Apple', 'Banana', 'Cherry', 'Durian', 'Figs'];

 @override
 Widget build(BuildContext context) {
 final combinedList = [firstList, secondList, thirdList];
 return Scaffold(
 appBar: AppBar(
 title: Text(widget.title),
 ),
 body: Column(
 children: [
 Expanded(
 child: Center(
 child: CustomScrollView(
 reverse: true,
 slivers: [
 SliverList(
 delegate: SliverChildBuilderDelegate((context, index) {
 return reversedMessageList(combinedList[index]);
 }, childCount: combinedList.length)),
 ],
 ),
 ),
 ),
 TextField(
 style: Theme.of(context).textTheme.bodyText1,
 keyboardType: TextInputType.multiline,
 maxLines: 10,
 minLines: 1,
 textCapitalization: TextCapitalization.sentences,
 decoration: const InputDecoration(
 labelText: 'Message',
 ),
 ),
 ],
 ),
 floatingActionButton: FloatingActionButton(
 onPressed: () {},
 child: const Icon(Icons.add),
 ),
 );
 }

 Widget reversedMessageList(List messages) {
 //All messages sharing the same date, where date is displayed as StickyHeader
 return StickyHeader(
 overlapHeaders: false,
 content: Column(
 children: [
 for (int i = 0; i < messages.length; i++)
 Container(
 decoration: const BoxDecoration(
 color: Color(0xD9B2924A),
 borderRadius: BorderRadius.all(Radius.circular(20))),
 child: Padding(
 padding: const EdgeInsets.all(20),
 child: Text(messages[i])))
 ],
 ),
 header: Padding(
 padding: const EdgeInsets.all(5),
 child: Center(
 child: Container(
 decoration: const BoxDecoration(
 color: Color(0xD9590E98),
 borderRadius: BorderRadius.all(Radius.circular(20))),
 child: Padding(
 padding: const EdgeInsets.all(8),
 child: Text('Number of messages ${messages.length}'))))),
 );

 URL

 }
}

@slightfoot is it possible to reopen this issue? Many thanks.

A solution that works for me is to add `WidgetsBindingObserver` mixin to the widget encapsulating the list view, add it as an observer to `WidgetsBinding.instance` and implement `didChangeMetrics` as follows:

dart
@override
 void didChangeMetrics() {
 super.didChangeMetrics();
 setState(() {});
 }

hope this helps


Question: Test dicovery doesn't provide feedback on failure?

Hello, thank you for all your patience and assistance- your help is critical to us being able to deliver the best product to our users! A few things:

1. Can you try switch your log level to trace (you can do so in the command palette) and then running it again and sending me both the logs from the `python` output channel and the `python test output` channel? I have extra error logging I've been adding and it might give some more insight.
2. To clarify on your setup it is a multiroot workspace with multiple modules per workspace? When you talk about projects A,B,C and D are these within a specific workspace? Then where does the 40+ modules come into play?
3. I see from that output you have a few plugins in use, does removing all of those change anything? I know you tried with the pytest-fixtures one already. `plugins: dictsdiff-0.5.8, pyfakefs-4.7.0, requests-mock-1.9.3, unordered-0.5.2, mock-3.10.0, cov-2.12.1, pytest_check-1.0.2, smtpdfix-0.4.2`

Thank you!

Test dicovery doesn't provide feedback on failure
I find quite often my tests "disappear" from the test window, or discovery appears to do nothing. This is almost always because I've introduced a syntax error in the package I'm testing which results in an import failure. The reason is clear in the Python Test Log, i.e. below I forgot to install a dependency culminating in discovery failing with exit code 2. If this should happen at the very least the test sidebar should provide some indication that test discovery exited abnormally - perhaps with a highlight on the "Show Test Output" icon since clicking on that is the first step in resolving the issue.

Test Discovery failed:
Error: ============================= test session starts ==============================

 import ahocorasick # pip install pyahocorasick
E ModuleNotFoundError: No module named 'ahocorasick'
=========================== short test summary info ============================
ERROR tests/test_label_model.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== 5 tests collected, 1 error in 2.00s ======================

Traceback (most recent call last):
 File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/run_adapter.py", line 22, in
 main(tool, cmd, subargs, toolargs)
 File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
 parents, result = run(toolargs, **subargs)
 File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
 raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

Hello! We have just finished our testing rewrite and are beginning the roll out to users. I have tested this issue with the re-write and I am no longer reproducing the bug! If you would like to try it yourself, you need to add this setting to your users `settings.json` `"python.experiments.optInto": ["pythonTestAdapter"]`. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!

Hi @eleanorjboyd, is there something else we need to do for this setting to take effect?

image

Actually, it looks like it _kind of_ works after putting the setting in my WSL config.

image

You can confirm you have the rewrite enabled by setting `"python.analysis.logLevel": "Trace",` in your user settings then check for `Experiment 'pythonTestAdapter'` is active in your python logs. Do you see that in your logs?

Looks like it is.

image

Here are my observations so far:

- The test extension takes a few seconds to appear on the left. It's the last one to "load."
- When I click on it, I first see this:
image
- A couple seconds later I see what's in my prior screenshot.

By the way, I selected to install the pre-release version of the Python extension. I'm not sure whether that's needed to test the `pythonTestAdapter` change. I was running the release version prior to this.

Hi! Could you send me your logs? I am wondering if there is an error there about UUIDs and if so would be connected to this: URL

Test discovery (pytest) fails in multi-root workspace: "payload UUID is not recognized"
Type: Bug

# Behaviour

It looks like test discovery was updated in the most recent Python release and it now breaks for my multi-root, multi-venv project.

When I run discovery, I see a stream of log messages about running the pytest commands and then a lot of messages like:

2023-07-10 14:31:48.395 [info] Using result resolver for discovery
2023-07-10 14:31:48.395 [info] Using result resolver for discovery
2023-07-10 14:31:48.395 [info] Using result resolver for discovery
2023-07-10 14:31:48.438 [error] On data received: Error occurred because the payload UUID is not recognized
2023-07-10 14:31:48.438 [error] On data received: Error occurred because the payload UUID is not recognized
2023-07-10 14:31:48.438 [error] On data received: Error occurred because the payload UUID is not recognized
2023-07-10 14:31:48.680 [info] Test server connected to a client.
2023-07-10 14:31:48.686 [info] Test server connected to a client.
2023-07-10 14:31:48.686 [info] Test server connected to a client.

Once the refresh completes, I'll see only one, sometimes two or my test packages int he explorer and nothing else.

It looks to me like you may have changed how tests are discovered and handled by the new test server (experiment is enabled) and the discovery processes are not communicating back to the thread that's expecting them? Or maybe only the UUID from the last spawned process is tracked so that's the only one that gets discovered?

## Expected vs. Actual

Expected: All of my tests to be discovered, displayed, and runnable from the test explorer.

Actual: Only tests from one module show up in test explorer.

## Steps to reproduce:

1. Create a multi-root workspace (mine has ~40 projects, 2 might be enough to trigger this?) where each project has it's own virtual environment in ` /.venv` and `python.defaultInterpreterPath: "${workspaceFolder}/.venv/bin/python`.
1. Define pytest tests in each of the projects
1. Open the Test Explorer view
1. Refresh tests if they are not auto-discovered

# Diagnostic data

- Python version (& distribution if applicable, e.g. Anaconda): 3.9.16
- Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
- Value of the `python.languageServer` setting: Default

`Python` output included in separate comment. Issue body too long.

Extension version: 2023.12.0
VS Code version: Code 1.80.0 (660393deaaa6d1996740ff4880f1bad43768c814, 2023-07-04T15:06:02.407Z)
OS version: Windows_NT x64 10.0.19044
Modes:
Remote OS version: Linux x64 3.10.0-1160.88.1.el7.x86_64 (**test discovery running on this remote**)
Remote OS version: Linux x64 4.18.0-425.19.2.el8_7.x86_64

 System Info

|Item|Value|
|---|---|
|CPUs|AMD EPYC 7763 64-Core Processor (8 x 2445)|
|GPU Status|2d_canvas: unavailable_software canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: disabled_software multiple_raster_threads: enabled_on opengl: disabled_off rasterization: disabled_software raw_draw: disabled_off_ok video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: unavailable_software webgl2: unavailable_software webgpu: unavailable_software|
|Load (avg)|undefined|
|Memory (System)|31.95GB (12.54GB free)|
|Process Argv|--crash-reporter-id 87e5818c-cbab-4d10-bc7f-19efe79b03de|
|Screen Reader|yes|
|VM|0%|

|Item|Value|
|---|---|
|Remote|SSH: remote.host.com (**test discovery running on this remote**)|
|OS|Linux x64 3.10.0-1160.88.1.el7.x86_64|
|CPUs|Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz (4 x 2499)|
|Memory (System)|30.81GB (17.23GB free)|
|VM|0%|

|Item|Value|
|---|---|
|Remote|SSH: remote2.host.com|
|OS|Linux x64 4.18.0-425.19.2.el8_7.x86_64|
|CPUs|Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz (4 x 2500)|
|Memory (System)|30.75GB (18.92GB free)|
|VM|0%|

 A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931:30280409
vshan820:30294714
vstes263:30335439
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263:30784847
vscaat:30438848
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
2e4cg342:30602488
pyind779:30671433
89544117:30613380
pythonsymbol12:30671437
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
03d35959:30757346
pythonfmttext:30731395
pythoncmv:30756943
fixshowwlkth:30771522
hideindicator:30785051
pythongtdpath:30769146
i26e3531:30780429
pythonnosm12tcf:30779713
pythonidxpt:30784022
pythonnocebcf:30776496
h7j2d465:30786200

Also, I can't seem to turn any of the python experiments off, no matter what I do in the config file(s). Not sure if that's by design or a completely separate bug.

Hello! Looking into these bugs you reported, I appreciate all your help! In terms of the experiment does ` "python.experiments.optOutFrom": ["pythonTestAdapter"],` not work for you? Did you reload the window? If that is not working then it is a different issue so let me know. Thanks!

`optOutFrom` works! Didn't realize that was a thing. Turning off the experiment gets my tests discoverable again, so that's a win. I'll check out the multi-root project on RHEL8 too and comment there if the symlink problem resolves the issue.

Looking into the UUID thing- one issue I have is I have trouble reproing it. Were you able to consistently get the error message `2023-07-10 14:31:48.438 [error] On data received: Error occurred because the payload UUID is not recognized`?

Yep. It comes up every time. I'll see if I can put together an example repo I can share, but not 100% I'll be able to do that easily.

I created a sample repo you can use. When I run this on my Windows VSCode with a Linux Remote over SSH, I get the UUID errors. When I run it on my MacBook, I don't get the UUID errors but I still get the same behavior; only one project's tests show up in the explorer. Not sure why the error messages aren't showing up on the Mac and haven't tried testing locally in Windows. Hopefully this helps you find a solution!

 URL

I'm actually running into "Error occurred because the payload UUID is not recognized" in a single workspace scenario (`pypa/packaging`).

We figured out the root cause- its a problem with when UUIDs are being deleted when the command to the test server is done running but not when the test server is done sending the response back. Therefore the UUID is being deleted pre-maturely. Working on a fix to have the actions to happen in the correct order.

I'm desperate! Using Vscode 1.80.1 and if I put:

"python.experiments.optOutFrom": ["pythonTestAdapter"]

It gets greyed out and I see this message:

<img width="981" alt="Screenshot 2023-07-15 at 12 10 10" src=" URL

What in heaven does it mean!?

I can't access my tests!!!

Well, well, I had to add this line specifically in to remote-server setting.

And added that actually: `"python.experiments.enabled": false,`

This is not great. I really want to centralise all my settings in one place and make exceptions... well, when they are **exceptions**!

Now I need to keep my eyes in **every** remote setup I have!

I simply don't want any experimental stuff! You guys have created the Insider Vscode for what then?

If you ever find yourself in a situation where the latest extension version simply is not working, you can pin to an older version that does work for you until we manage to fix the bug causing you issues.

I wouldn't label this work as experimental. The new testing code passed through pre-release users and internal team testing. It's unfortunate this bug didn't trigger until more people were able to use the new code in more varied scenarios. It's behind an experiment flag so we can control the rollout in case of showstopper bugs, not because we are actively "experimenting" with our stable users (as I said, this code already passed various quality gates to even reach this level of rollout). If we had not used our experiment infrastructure to control this then this would have gone out to all users this release with no possibility for us to do an emergency rollback without pushing a new version of the extension or given you a way to opt out in any way until we made a new release with a fix.

To get early feedback from users willing to help us find bugs early. Obviously this didn't come up from testing with our pre-release users and are now only running across this from wider usage.

How to pin?

I apologise for my ton but when one wastes a whole morning, in a weekend with a critical deadline...
I'm not against experiments but, please, make it easier to opt-out.

You can change the version in the extension view via "Install Another Version":

image

Now if you want it to update when we push a new stable version, that's all you need to do.

If you want to stay on that version, you can turn off automatic updates for the extension:

image

Otherwise you need to turn off automatic extension updates via the `extensions.autoUpdate` setting (unfortunately it's a global setting).

Apology accepted. We realize hitting any bug is annoying and we obviously do what we can to avoid it; this is why there isn't anything the extension does that you couldn't do manually (e.g., run your test suite from the command-line instead of having VS Code do it). We always strive to make sure there's no lock-in specifically to make sure we are never a complete blocker for one's work or you feel trapped by the extension.

@eleanorjboyd I'm assuming this fix is in the latest version of the plugin? I'm still not seeing all my tests discovered. Haven't tried against the skeleton repo I sent you yet, but our actual project is only showing me one test package.

Also, it no longer shows all my subpackages as top-level elements in the test view. Instead, I have the root of our project and have to drill down our folder structure to see the tests. With the multi-root worksapce, I would still expect to see each root module as a test parent.

Still seeing the same issue with Insiders build and pre-release versions of Python/Pylance.

Hi, are you able to check if this is now resolved on insiders? I have made even more edits to the flow to hopefully not delete the UUID pre-maturely which I think was the problem. If not I will have to go a different direction for the investigation. Thanks for your help!

Problem is still there. I'm getting a ton of these errors:

 The python test process was terminated before it could exit on its own, the process errored with: Code: 5, Signal: null
2023-10-05 13:04:21.392 [error] pytest test discovery error

Once the test refresh finishes, I see the tests for one of my projects, a `pytest Discovery Error` for another that references the error messages above, and none of the other projects in the workspace. If I refresh again, the view clears and a few different projects flicker in the list as their tests get collected, but they're immediately replaced by whatever finishes next and I ended up with the view I had originally.

So... simplified version, if my workspace has projects A, B, C, and D and I refresh tests, the Test Explorer shows:

- A
-- tests
--- unit
---- test_xyz
- pytest Discovery Error [B]

If I refresh tests again, I'll see the tests for C appear instead of A for about a second, then discovery completes on A and I'm back to the view above. It does look like the Discovery Error bounces between B and D with every refresh.

I am able to run the tests for A once the refresh completes.

@mrburrito just to make sure, do you have any other pytest plug-ins configured?

Only one is `pytest-fixtures` which I wouldn't think would interfere with test discovery, but I'll turn it off and see if anything changes.

Local plugins:

DavidAnson.vscode-markdownlint@0.52.0
dsznajder.es7-react-js-snippets@4.4.3
eamodio.gitlens@2023.6.605
ecmel.vscode-html-css@1.13.1
foxundermoon.shell-format@7.2.5
GitHub.vscode-pull-request-github@0.75.2023100509
hashicorp.terraform@2.29.2023100414
iliazeus.vscode-ansi@1.1.6
k--kato.intellij-idea-keybindings@1.5.10
lizebang.bash-extension-pack@0.0.4
mads-hartmann.bash-ide-vscode@1.39.0
marclipovsky.string-manipulation@0.5.7
ms-azuretools.vscode-docker@1.26.1
ms-python.black-formatter@2023.5.11881008
ms-python.isort@2023.11.11881009
ms-python.python@2023.19.12791027
ms-python.vscode-pylance@2023.10.11
ms-vscode-remote.remote-containers@0.317.0
ms-vscode-remote.remote-ssh@0.106.4
ms-vscode-remote.remote-ssh-edit@0.86.0
ms-vscode-remote.remote-wsl@0.81.7
ms-vscode-remote.vscode-remote-extensionpack@0.24.0
ms-vscode.powershell@2023.6.0
ms-vscode.remote-explorer@0.4.1
ms-vscode.remote-server@1.4.3
nickmillerdev.pytest-fixtures@0.3.3
qux-bbb.text-utils@0.1.7
redhat.ansible@2.7.98
redhat.vscode-yaml@1.14.0
Remisa.shellman@5.7.0
rogalmic.bash-debug@0.3.9
rpinski.shebang-snippets@0.1.4
timonwong.shellcheck@0.34.0

Remote Plugins:

DavidAnson.vscode-markdownlint@0.52.0
dsznajder.es7-react-js-snippets@4.4.3
eamodio.gitlens@14.3.0
ecmel.vscode-html-css@1.13.1
foxundermoon.shell-format@7.2.5
GitHub.vscode-pull-request-github@0.74.0
hashicorp.terraform@2.28.0
iliazeus.vscode-ansi@1.1.6
lizebang.bash-extension-pack@0.0.4
mads-hartmann.bash-ide-vscode@1.39.0
marclipovsky.string-manipulation@0.5.7
ms-azuretools.vscode-docker@1.26.1
ms-python.black-formatter@2023.4.1
ms-python.flake8@2023.9.12791029
ms-python.isort@2023.10.1
ms-python.python@2023.19.12791027
ms-python.vscode-pylance@2023.10.11
ms-vscode.powershell@2023.6.0
nickmillerdev.pytest-fixtures@0.3.3
qux-bbb.text-utils@0.1.7
redhat.ansible@2.7.98
redhat.vscode-yaml@1.14.0
Remisa.shellman@5.7.0
rogalmic.bash-debug@0.3.9
rpinski.shebang-snippets@0.1.4
timonwong.shellcheck@0.34.0

Same behavior with that plugin enabled or disabled. I'm seeing a different set of tests today but still only one of my 40+ modules showing up.

Not sure if this is relevant or if the discovery code accounts for it, but the pytest discovery check is returning the same ` ` top-level element for almost all of my modules.

Sample structure:

/root_dir
 /packages
 /module1
 /module2
 /stages
 /module3
 /module4

Sample output:

============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.3.0
rootdir: /root_dir
plugins: dictsdiff-0.5.8, pyfakefs-4.7.0, requests-mock-1.9.3, unordered-0.5.2, mock-3.10.0, cov-2.12.1, pytest_check-1.0.2, smtpdfix-0.4.2
collected 10 items

========================= 10 tests collected in 1.50s ==========================
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.3.0
rootdir: /root_dir
plugins: dictsdiff-0.5.8, pyfakefs-4.7.0, requests-mock-1.9.3, unordered-0.5.2, mock-3.10.0, cov-2.12.1, pytest_check-1.0.2, smtpdfix-0.4.2
collected 10 items

========================= 10 tests collected in 1.50s ==========================
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.3.0
rootdir: /root_dir
plugins: dictsdiff-0.5.8, pyfakefs-4.7.0, requests-mock-1.9.3, unordered-0.5.2, mock-3.10.0, cov-2.12.1, pytest_check-1.0.2, smtpdfix-0.4.2
collected 10 items

========================= 10 tests collected in 1.50s ==========================
============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.3.0
rootdir: /root_dir
plugins: dictsdiff-0.5.8, pyfakefs-4.7.0, requests-mock-1.9.3, unordered-0.5.2, mock-3.10.0, cov-2.12.1, pytest_check-1.0.2, smtpdfix-0.4.2
collected 10 items

========================= 10 tests collected in 1.50s ==========================

That output doesn't seem to have any bearing on what gets displayed in the test explorer, though I haven't seen any of the modules with fully qualified paths show up.

Oops. Somehow missed that last comment. I'll respond with details in a few minutes.

1. I'll try to bump logging and attach. If I can reproduce with my bare bones example project, it'll be pretty straightforward, otherwise I need to do some log cleanup and might take a bit.

2. I may be confusing things with terminology. I have a multi-root workspace with ~60 defined roots. All but 5-6 of those have their own Python environment defined with pytest tests. We do have them organized into smaller subsets (e.g. stages, packages, etc. as I described above), but from a VSCode view, everything is on one level, so my roots might be:

 packages :: A (./packages/A)
 packages :: B (./packages/B)
 stages :: C (./stages/C)
 stages :: D (./stages/D)

 When I'm refreshing tests, I might see the tests for `packages :: A`, then refresh and see only the tests for `stages :: C`, but never all of the tests.

3. Those are pytest plugins, not VSCode extensions. Can't really disable them because they're in use by our tests.

That test run is from this example repository. It replicates the behavior I see on my real project and answers your question 3 above since it doesn't use any pytest plugins. It's also almost identical in structure to my real repo (question 2), though that's grown by a few roots since I created the example.

 URL

Hello, thank you for the detailed logs and all the notes I really appreciate it. I am going out of office for the next 3 weeks and therefore will not be able to help in this time. I will loop in @karthiknadig in case you have outstanding questions to get by in the meantime.

Additional, @karthiknadig do you know what this log is saying / if it might impact?

2023-11-06 14:16:30.540 [warning] Failed to check if /alt/source/vscode-test-discovery/group1/group1_p3/.venv/bin/python is an executable [Error: ENOENT: no such file or directory, lstat '/alt/source/vscode-test-discovery/group1/group1_p3/.venv/bin/python'] {
 errno: -2,
 code: 'ENOENT',
 syscall: 'lstat',
 path: '/alt/source/vscode-test-discovery/group1/group1_p3/.venv/bin/python'
}

So, several of our projects don't have Python environments but it seems like the Python plugin tries to find the virtualenv anyway. I left that configuration in the example so it mirrored my actual project. Those are the Node errors that get thrown when it tries to find `${workspaceRoot}/.venv/bin/python` and it doesn't exist.

realized now this isn't the problem- seeing other messages elsewhere which show it did run and this was just a warning.

New thought actually, @karthiknadig what do you think about the theory that in this multiroot workspace case the results are attempting to be parsed by the wrong adapter (like an adapter from a different workspace) instead of the one which it actually correlates to. Therefore it would error out all the other workspaces instead of just letting them wait for their response to be returned. Thinking `this._onDataReceived.fire({ uuid: '', data: '' });` this line here if deleted might lead to success since it doesn't cause unsuccessful resolves on other workspaces. But I am unsure if I am getting the multiroot scenario correctly @karthiknadig for this to be true

That makes sense. This should get addressed when we use the per-run named-pipe server.

I have the exact same problem, let me know if I can provide some feedback

update for those in this thread with symlink issues:

I have put together a detailed problem report and proposed solution for symlinks. Please check it out here and contribute any thoughts you may have: URL

Is the symlink fix expected to fully resolve this or is that just a related fix? I think I'm still having this problem even if I open it from the non-symlinked path (which I've been doing because of all the symlink problems!).

I'm no longer on the team that was having the issues. Let me try to get somebody to test it out and get back to you! Looking to see if I can reproduce on a different project as well.

@alanwilter I have re-opened the issue so we can discuss there- thanks

Also closing this issue since we dont have the repro anymore but please create another issue / follow up on #22018 about symlinks so we can discuss any ongoing problems


Question: Error deploying Wazuh Indexer cluster via Puppet?

I did a test installation with the classes for distributed installation and found several problems:

It was necessary to declare each of the nodes and they were fixed, so I opted to modify the code of the `indexer_dist.pp` class so that it does not configure any parameters of the `opensearch.yml` template except the default values and that they are assigned in the installation manifest

The config.yml file for the creation of the certificates has fixed values assigned and I am investigating how to be able to assign these values of each node by means of an array, which should have a pair of values for each item (node name, ip) So I continue with the investigation to review how to assign these values to the creation template of the `config.yml` file.

Modifying the `indexer_dist.yml` manifest allowed me to deploy an AIO environment, but it requires a different configuration of the install manifest than what we currently have:

$discovery_type = 'single'
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-35-35.us-east-2.compute.internal" {
 class { 'wazuh::certificates_dist':
 stage => certificates,
 }
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
 stage => indexerdeploy,
 }
 class { 'wazuh::manager':
 manage_repos => false,
 stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
 stage => manager,
 }
 class { 'wazuh::dashboard_dist':
 stage => dashboard,
 }
}

Error deploying Wazuh Indexer cluster via Puppet
|Wazuh version|Component|Install type|Install method|Platform|
|---|---|---|---|---|
| 4.6.0 Beta 1 | Wazuh component | Indexer | Puppet | Ubuntu 18.04|

I discovered in the test carried out in this issue that, deploying indexer cluster via puppet isn't successful. Each indexer node is installed and configured separately but are not part of the same cluster.

Testing the Wazuh indexer cluster with the security admin script provides the output below showing only one node:

console
root@indexer2:/home/rolly# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** URL **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.8.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
 SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
 SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
 SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
 SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
 SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
 SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
 SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
 SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
 SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml
 SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
Done with success

Also although dashboard deployment was not part of the test description, I however tried to install but got the following error when trying to access via URL.

image

The dashboard error log shows the following:

console
root@dashboard:/home/rolly# tail /tmp/wazuh-dashboard.log
Oct 04 11:33:15 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:15Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:17 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:17Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:20 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:20Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:22 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:22Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:25 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:25Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:27 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:27Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:30 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:30Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:32 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:32Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:35 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:35Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Oct 04 11:33:37 dashboard opensearch-dashboards[6486]: {"type":"log","@timestamp":"2023-10-04T11:33:37Z","tags":["error","opensearch","data"],"pid":6486,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

The dashboard searches for the indexer on `localhost` instead of its IP address(`192.168.21.170` or `192.168.21.171` in the case of this test)

Currently, the wazuh-puppet repository does not allow a distributed deployment.
It will be included in `4.8.0` with this issue:
- URL

We will continue the distributed deployment development here:
- URL

Add option for distributed deploy
In the documentation of Wazuh Puppet deploy only have an example for deploy an AIO and it is necessary add another example about deployment with distributed servers.

An analysis was made of the changes that are required on the wazuh-puppet repository in order to deploy a distributed environment:
Modify the use of the template for the config.yml. The config.yml file that is currently used is a fixed template that only contains this data:

 yaml
# config.yml
nodes:
 indexer:
 -name: indexer
 ip: 127.0.0.1
 server:
 - name: server
 ip: 127.0.0.1
 dashboard:
 - name: dashboard
 ip: 127.0.0.1

It is necessary to modify the wazuh::certificates class so that when creating the certificates it creates a config.yml file according to the environment that is going to be deployed.

Modify the movement of the `.pem` and `key.pem` keys of each node. These files are created on the same server where Wazuh indexer is going to be deployed, so when moving these files it only moves them to the corresponding directories. This process is generated with hard coded data, since just as the config.yml file is fixed, so are the names of the keys and that is why it performs a move of the exact files, without taking into account variables and so on.

In itself, the dynamics of the current AIO and the distributed environment are very different, since each current class of each product is almost independent, so at the time of installation they do not require previous steps as if the distributed environment requires and it is extremely necessary that the certificates of each domain are generated in one location and then moved, or at least a unique root-ca is generated. Then each node creates its certificates with this root-ca in common, which is currently not contemplated and is done in a very simple way on a single server.

A new manifest was created where the wazuh::indexer class is declared, which mainly removes many hardcoded options and allows using variables for indexer deployment with more than one node, such as the list of nodes to install.

A new template of the config,yml file was created, to be able to assign variables with the names and IPs of all the nodes for which a certificate is going to be generated. The previous template was too basic and was hardcoded with the names and IPs that belonged to localhost, the new template contemplates different variables assignable by the deployment manifest:

 yaml
nodes:
 indexer:
 - name: indexer-
 ip:
 - name: indexer-
 ip:
 - name: indexer-
 ip:
 server:
 -name: manager-master
 ip:
 node_type: master
 -name: manager-worker
 ip:
 node_type: worker
 dashboard:
 - name: dashboard
 ip:

A new opensearch.yml template was created since the previous one had the names of all the node certificates hardcoded, so it is necessary to change these values.

The section where the securityadmin was executed, which only has to be run on the main node of the cluster, was removed from within the indexer manifest, and a new class was created to perform this action later.

We worked on a new deployment script, it is not finished but its objective is to be able to assign the necessary variables for the deployment beforehand, such as the names of the nodes and their IPs:

 ruby
$node1-ip =
$node2-ip =
$node3-ip =
$master-ip =
$worker-ip =
$dashboard-ip =
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'

class['wazuh::certificates-dist']

node "node1" {
 if $facts['os']['family'] == 'Debian' {
 Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
 } else {
 Class['wazuh::repo'] -> Package['wazuh-indexer']
 }
 class { 'wazuh::indexer-dist':
 indexer_node_name => "$indexer_node1_name",
 indexer_network_host => "$node1-ip",
 }
}
node "node2" {
 if $facts['os']['family'] == 'Debian' {
 Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
 } else {
 Class['wazuh::repo'] -> Package['wazuh-indexer']
 }
 class { 'wazuh::indexer-dist':
 indexer_node_name => "$indexer_node2_name",
 indexer_network_host => "$node2-ip",
 }
}
node "node3" {
 if $facts['os']['family'] == 'Debian' {
 Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
 } else {
 Class['wazuh::repo'] -> Package['wazuh-indexer']
 }
 class { 'wazuh::indexer-dist':
 indexer_node_name => "$indexer_node3_name",
 indexer_network_host => "$node3-ip",
 }
}

Tests must be done and verification of running securityadmin at the end of the installation and subsequent transfer of the previously created certificates.

The configuration of the distributed environment of Wazuh through Puppet is finished.

For the installation, the following manifest must be executed, adding the names of the Puppet agent nodes to each one and adding the necessary IPs.

 rb
$node1host = '172.31.2.205'
$node2host = '172.31.12.65'
$node3host = '172.31.6.144'
$masterhost = '172.31.2.205'
$workerhost = '172.31.12.65'
$dashboardhost = '172.31.6.144'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}

node "ip-172-31-18-215.us-west-1.compute.internal" {
 class { 'wazuh::certificates_dist':
 stage => certificates,
 }
}
node "ip-172-31-2-205.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
 indexer_node_name => "$indexer_node1_name",
 indexer_network_host => "$node1host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
 class { 'wazuh::manager':
 ossec_cluster_name => 'wazuh-cluster',
 ossec_cluster_node_name => 'wazuh-master',
 ossec_cluster_node_type => 'master',
 ossec_cluster_key => '01234567890123456789012345678912',
 ossec_cluster_bind_addr => "$masterhost",
 ossec_cluster_nodes => ["$masterhost"],
 manage_repos => false,
 ossec_cluster_disabled => 'no',
 stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
 filebeat_oss_indexer_ip => "$node1host",
 stage => manager,
 }
}
node "ip-172-31-12-65.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
 indexer_node_name => "$indexer_node2_name",
 indexer_network_host => "$node2host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 stage => indexerdeploy,
 }
 class { 'wazuh::manager':
 ossec_cluster_name => 'wazuh-cluster',
 ossec_cluster_node_name => 'wazuh-worker',
 ossec_cluster_node_type => 'worker',
 ossec_cluster_key => '01234567890123456789012345678912',
 ossec_cluster_bind_addr => "$masterhost",
 ossec_cluster_nodes => ["$masterhost"],
 manage_repos => false,
 ossec_cluster_disabled => 'no',
 stage => manager,
 }
}
node "ip-172-31-6-144.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
 indexer_node_name => "$indexer_node3_name",
 indexer_network_host => "$node3host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 stage => indexerdeploy,
 }
 class { 'wazuh::dashboard_dist':
 indexer_server_ip => "$node1host",
 manager_api_host => "$masterhost",
 stage => dashboard,
 }
}

The manifest has several stages, in which the certificates are created in the Puppet server, they are moved to the directory of one of the modules, which will serve to later move the certificates to the necessary agents, then proceed to install Wazuh indexer on each node, securityadmin is run and then Wazuh manager (master and worker) and Wazuh dashboard are installed.

Made all documentation changes in the URL branch and created the PR URL pull/6186 for approval.

The commits of this change were taken to merge to the 4.8.0 branch, a complete deployment was tested with 3 `Wazuh indexer` nodes, 1 `Wazuh dashboard`, 1 `Wazuh manager` master and 1 `Wazuh manager` worker and the deployment was carried out successfully.

$node1host = '172.31.23.191'
$node2host = '172.31.28.121'
$node3host = '172.31.24.156'
$masterhost = '172.31.28.245'
$workerhost = '172.31.28.97'
$dashboardhost = '172.31.31.95'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
$indexer_discovery_hosts = [$node1host, $node2host, $node3host]
$indexer_cluster_initial_master_nodes = [$node1host, $node2host, $node3host]
$indexer_cluster_CN = [$indexer_node1_name, $indexer_node2_name, $indexer_node3_name]
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
stage { 'agent': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard] -> Stage[agent]
Exec {
 timeout => 0,
}
node "ip-172-31-26-67.us-west-1.compute.internal" {
 class { 'wazuh::certificates':
 indexer_certs => [["$indexer_node1_name","$node1host" ],["$indexer_node2_name","$node2host" ],["$indexer_node3_name","$node3host" ]],
 manager_master_certs => [['master',"$masterhost"]],
 manager_worker_certs => [['worker',"$workerhost"]],
 dashboard_certs => ["$dashboardhost"],
 stage => certificates,
 }
 class { 'wazuh::repo':
 stage => repo,
 }
}
node "ip-172-31-23-191.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
 indexer_node_name => "$indexer_node1_name",
 indexer_network_host => "$node1host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 indexer_discovery_hosts => $indexer_discovery_hosts,
 indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
 indexer_cluster_CN => $indexer_cluster_CN,
 stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
}
node "ip-172-31-28-121.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
 indexer_node_name => "$indexer_node2_name",
 indexer_network_host => "$node2host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 indexer_discovery_hosts => $indexer_discovery_hosts,
 indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
 indexer_cluster_CN => $indexer_cluster_CN,
 stage => indexerdeploy,
 }
}
node "ip-172-31-24-156.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
 indexer_node_name => "$indexer_node3_name",
 indexer_network_host => "$node3host",
 indexer_node_max_local_storage_nodes => "$cluster_size",
 indexer_discovery_hosts => $indexer_discovery_hosts,
 indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
 indexer_cluster_CN => $indexer_cluster_CN,
 stage => indexerdeploy,
 }
}
 node "ip-172-31-28-245.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
 ossec_cluster_name => 'wazuh-cluster',
 ossec_cluster_node_name => 'wazuh-master',
 ossec_cluster_node_type => 'master',
 ossec_cluster_key => '01234567890123456789012345678912',
 ossec_cluster_bind_addr => "$masterhost",
 ossec_cluster_nodes => ["$masterhost"],
 ossec_cluster_disabled => 'no',
 stage => manager,
 }
}
 node "ip-172-31-28-97.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
 ossec_cluster_name => 'wazuh-cluster',
 ossec_cluster_node_name => 'wazuh-worker',
 ossec_cluster_node_type => 'worker',
 ossec_cluster_key => '01234567890123456789012345678912',
 ossec_cluster_bind_addr => "$masterhost",
 ossec_cluster_nodes => ["$masterhost"],
 ossec_cluster_disabled => 'no',
 stage => manager,
 }
}
node "ip-172-31-31-95.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::dashboard':
 indexer_server_ip => "$node1host",
 manager_api_host => "$masterhost",
 stage => dashboard,
 }
}

In addition, the documentation was modified so that the distributed example does not try to install Wazuh agent on the nodes and the commits were moved to merge to the `4.8.0` branch of `wazuh-documentation`.


Question: TransformerPS1S2 Issue?

I have uploaded a new `Transformer.lib` library. Thanks for the contribution! Closing this.

TransformerPS1S2 Issue
I'm having issues trying to get a 6-phase rectifier circuit to work. I sent the netlist over to the ngspice guys and they found a possible error with .SUBCKT Transformers_TransformerPS1S2. The netlist line "R3 nSPlus nSplus {RS1}" is wrong. There may be other errors in the .SUBCKT.

 URL

2023-11-14_105159

6_Phase_Rectifier.zip

**TODO** list:
- [x] Fix short circuit in transformer model
- [x] Add TD parameter for sine source
- [x] Deprecate red AC source

Holger recommendation is as follow. I cannot find "chgtol" in the Qucs settings.

You may add this line to the netlist.
.options abstol=10u chgtol=1e-10 method=gear
 Because you have high voltages, chgtol and abstol reduce the precision requirements, especially the gear method here yields a transient solution. It looks strange though, and you should simulate up to 1 s to see the output signal decay.

Yes, the options helps. You may use the .OPTIONS device.
image

Output is wrong. Supposed to look like the following.
6-phase-rectifier

6-phase-rectifier-out

Holger said to use Delay instead of Phase for the AC source. This and the changes to Transient Parameters seems to solve the issues.

Can the "Qucs" AC source have VO (offset) and TD (delay) added as a parameter? The "Spice" AC source usage is a bit confusing.

Attached is the "good and bad" schematics. As shown below, when using Delay in the AC source, the 120 and 240 Degree transformer outputs had a +/-200 volt DC offset. Unsure how this is possible. The offset occurs even if the diodes are removed.

Waiting for Holger to verify if this is a bug.

2023-11-14_152103

6_Phase_Rectifier.zip

Yes, you may use the generic SPICE source for this purpose:
image

No, this is done intentionally for Qucs compatibility. This node never caused problems before.

The "ac Voltage Source" at the bottom of the Sources menu is a Spice only component so Qucs compatibility isn't an issue. It has a single default entry of "Vac" which is wrong. ngspice thinks it is a DC source. Why was this device created when "V Source" is available.

Can you alter "ac Voltage Source" by adding all the parameters to the Menu so the User can change them?

2023-11-15_111458

2023-11-15_111955

2023-11-15_113042

2023-11-15_113155

Yes, I have just added the TD and VO parameters for AC source. This fix will be available since the new release.

image

The "ac Voltage Source" at the bottom of the Sources menu will still be wrong if not changed or deleted. Why was it created?

2023-11-15_115150

This device was added by @MikeBrinson long time ago. I cannot say what was the purpose for this device. I am also finding two type of source confusing. I may delete it in the next release. The *V source* device would be sufficient if one need full SPICE definition.

I have deprecated the red AC source. The remaining issues with the transformer library will be fixed in #367 Closing this as completed.

XSPICEPS1S2 transformer issues
I found more problems with XSPICEPS1S2 in Transformer.lib. I made three circuits comparing XSPICEPS1S2 and TransformerPS1S2 in Transformer.lib to Three Mutual Inductors in Components. Analysis of TransformerPS1S2 and Three Mutual Inductors generally agree but XSPICEPS1S2 is wrong and generates errors.

XSPICEPS1S2 has two issues:

1. Rp, Rs1 and Rs2 generate warnings. Is there node naming issues like in TransformerPS1S2?
2. The "Turns Ratio" is half what it should be. You can see the output voltages and currents in the plots are half compared to transformers TransformerPS1S2 and Three Mutual Inductors.

I recreated the subcircuit for XSPICEPS1S2 made by Mike Brinson. It can be modified to fix the "turns ratio" problem.

I also recreated the subcircuit for XSPICEPS1. It's "turns ratio" seems OK.

**TODO**

- [x] Replace models in Transformer.lib

2023-11-15_192246

2023-11-15_205939

2023-11-15_215510

Transformer_Lib_Test_prj.zip

No, the variable should be wrapped in braces like `{Rs1}` Otherwise Ngspice may interpret it as the model.

Holger corrected the XSPICEPS1S2 subcircuit. AFAIK, the info in the online help is wrong.

I found another issue, Cores.lib uses the same data for every core so only one core is correct....

Yes, I have also noticed that the Cores.lib uses duplicate data. Unfortunately, I have no source data for this library and cannot provide the fix. The library was developed by @MikeBrinson around 2015. I cannot say where did he take the cores data.

The help on readthedocs.io should be deprecated and replaced by the new tutorial. It contains a lot of outdated information. I have started to collect the new documentation here: URL But this a slow process. Writing documentation requires much more time effort than writing a code.

Here is the updated Transformers.lib. All identified errors have been fixed.

I left the "Cores" data in Transformers.lib and Cores.lib as it was. I could not find any info on the web. Looks like manually pulling data from B/H plots would be required. Some of the cores are nothing a user would ever make a transformer.

I requested info from Mike Brinson...

Transformers.zip


Question: Terminus error -- call to undefined method?

I got some reports in Slack of this working so I'm closing this ticket. Thanks everyone involved!

Feel free to open a new issue if there is something else that needs to be addressed :)

Terminus error -- call to undefined method
Good day,
We got the error as following when trying to build the environments thru GitHub action:
`PHP Fatal error: Uncaught Error: Call to undefined method Pantheon\TerminusBuildTools\Commands\EnvPushCommand::getsite() in /home/runner/.terminus/terminus-dependencies-c788000802/vendor/pantheon-systems/terminus-build-tools-plugin/src/Commands/BuildToolsBase.php:1120`

It is PR from `mylocalbranch` -> `develop` branch. The workflow file install terminus part as blowing:

 - name: Install Terminus
 run: |
 mkdir ~/terminus && cd ~/terminus
 curl -L URL --silent " URL | perl -nle'print $& while m{"tag_name": "\K.*?(?=")}g')/terminus.phar --output terminus
 chmod +x terminus
 sudo ln -s ~/terminus/terminus /usr/local/bin/terminus

I tried to downgrade teriminus 3.6.2 version, action still failed and same error as above.
`$(curl) --silent " URL
Please let me know if more info need to provided.

Thanks.

Image

This is happening when pushing to Pantheon multidev environments. Didn't try `master` branch yet but feel it will failed as well.

Duplicate of #497, a fix is linked there, we'll get that merged and a new release out.

terminus build:workflow:wait results in call to undefined method getsite() with Terminus 4.0
After upgrading to Terminus 4.0 and Terminus Build Tools Plugin 3.0.10, I get the following error when running `terminus build:workflow:wait`

It appears a reference to getSite() was missed in URL

 URL

We've also experienced our build process this morning on `terminus build:env:push`:

Image

Another fix was just merged and a new version was tagged. Please give it a try
Thanks!


Question: [fhir-client] Calling $validate from the fhir-client fails in the main branch?

Error 404 issued with the text the "unknow" text is not a valid resource Closing issue.
The 404 is given for both 4.0 and 4.3 fhir versions

[fhir-client] Calling $validate from the fhir-client fails in the main branch
We are using the FHIRClient validate method to validate Bundle resource by passing the Bundle object.

client.validate(finalBundle)

It fails to perform validation. The FHIR server log displays following message

[6/17/22, 16:27:24:361 UTC] 00000042 FHIRHttpServl 2 Creating FHIRHttpServletRequestWrapper for HttpServletRequest: com.ibm.ws.webcontainer40.srt.SRTServletRequest40@ee864739
[6/17/22, 16:27:24:361 UTC] 00000042 FHIRHttpServl 2 Retrieved these 'header' query parameters from the request URI: {}
[6/17/22, 16:27:24:361 UTC] 00000042 FHIRHttpServl 2 Content-Type is application/xml
[6/17/22, 16:27:24:361 UTC] 00000042 FHIRRestServl I Received request: tenantId:[default] dsId:[default] user:[fhiruser] method:[POST] uri:[ URL
[6/17/22, 16:27:24:365 UTC] 00000042 FHIRProvider > com.ibm.fhir.provider.FHIRProvider readFrom ENTRY
[6/17/22, 16:27:24:375 UTC] 00000042 FHIRProvider com.ibm.fhir.server.resources.Operation invoke(String,String,Resource) ENTRY
[6/17/22, 16:27:24:376 UTC] 00000042 FHIRResource 1 'Resource' is not a valid resource type.
[6/17/22, 16:27:24:381 UTC] 00000042 FHIRResource 1
OperationOutcome:
{"resourceType":"OperationOutcome","id":"ac-13-0-2-cd77fa72-e66c-4b97-a533-56c7af817a67","issue":[{"severity":"fatal","code":"not-supported","_code":{"extension":[{"url":" URL is not a valid resource type."}}]}
[6/17/22, 16:27:24:381 UTC] 00000042 RestAuditLogg > com.ibm.fhir.server.util.RestAuditLogger logOperation ENTRY
[6/17/22, 16:27:24:381 UTC] 00000042 RestAuditLogg com.ibm.fhir.provider.FHIRProvider writeTo ENTRY
[6/17/22, 16:27:24:385 UTC] 00000042 FHIRProvider < com.ibm.fhir.provider.FHIRProvider writeTo RETURN
[6/17/22, 16:27:24:385 UTC] 00000042 FHIRRestServl I Completed request[0.024226862 secs]: tenantId:[default] dsId:[default] user:[fhiruser] method:[POST] uri:[ URL status:[404]
[6/17/22, 16:27:24:385 UTC] 00000042 FHIRRestServl < com.ibm.fhir.server.filter.rest.FHIRRestServletFilter doFilter RETURN

**Following is a sample Bundl resource that has validation error but the validation operation does npt detect it**
EST-REG-ForVikas-RegTesting-Enriched.xml.zip

Thanks Vikas. It looks like the client was dependent on $validate being served at `[base]/Resource/$validate`.
We used to support that but I removed that support in #3265.
I think we'll need to update the client to infer the resource type from the resource that was passed.

 URL for more info.

@vikforfda what kind of object is your `finalBundle`?

our FHIRClientImpl validate implementation is already invoking validate on a type-specific basis, and so I now suspect this is an issue with your code and not ours.

The issue was on our side. We were using an older version of fhir client to invoke the validation operation API. After we updated it to use versions 4.11.1 (most recent IBM-FHIR release) or 5.0.0-SNAPSHOT (corresponds to the WIP IBM-FHIR R4b release) then the validation correctly identified the error

Remove any remaining support for requests to abstract endpoints like `[base]/Resource`
**Is your feature request related to a problem? Please describe.**
In URL we added warnings for requests to "abstract" type endpoints like `[base]/Resource`.
Now I'd like to pull the trigger and remove that support altogether.

**Describe the solution you'd like**
have requests to `[base]/Resource` handled just like requests to `[base]/bogus`

**Describe alternatives you've considered**
keep supporting it in whatever way we do today

**Acceptance Criteria**

1.
 GIVEN a 5.0 version of IBM FhIR Server
 WHEN a request is made to `[base]/bogus`
 THEN it results in a 404 not found

**Additional context**
one such place is FHIRResource.checkType(String):

if (!ModelSupport.isConcreteResourceType(type)) {
 log.warning("Use of abstract resource types like '" + type + "' in FHIR URLs is deprecated and will be removed in a future release");
}


Question: Update help pages to clarify ownership of requests once published?

Simplest thing to do here is change the default value of the select box and add more emphasis to the consequence of selecting a particular value:

<img width="702" alt="Screenshot 2022-09-21 at 14 38 53" src=" URL

<img width="685" alt="Screenshot 2022-09-21 at 14 38 59" src=" URL

Update help pages to clarify ownership of requests once published
As discussed on WDTK volunteer catch up call 24/06/2021 - URL

- Should make clear to pro and non pro users the requests aren’t “theirs” once published.
- Treat like publishing under Creative Commons / Open licence
- “You’re making this a public asset” .. “doing a public service”.
- Issue with existing requests .. can’t retrospectively apply a public licence.

At a minimum we need to clarify

* What circumstances might we hide material or destroy data on their requests
* Policies on re-embargoing. Need technical measures ( URL first, but even if they exist it needs to be clear in UI and/or help pages what the expectations should be.

URL would be a good place for this.

Need to also check T&C's.

This issue is being closed due to a lack of discussion or resolution for over 12 months. Should we decide to revisit this issue in the future, it can be reopened.

Make Pro requests public by default
Currently there is a default position of a 3 month embargo on Pro requests.

I suggest making no embargo the default, and requiring the setting of an embargo to be something which needs to be done actively, this would promote the publication of requests and responses.

Issue raised having seen a Tweet by a Pro user who appears unsure about the embargo they have set:

 URL

+1 I keep nearly making embargoed requests by mistake. It'd be good to be able to set a default that applied to all my requests unless I said otherwise.

Would need to consider interaction with URL / URL Just-created requests should have the same mechanics as just-published requests.

This issue has been automatically closed due to a lack of discussion or resolution for over 12 months.
Should we decide to revisit this issue in the future, it can be reopened.