Binary patching rust hot-reloading, sub-second rebuilds, independent server/client hot-reload (#3797)
Enable runtime rust hot patching. * roll back wry to 0.45 * cleanup rustc * wip * wip: fix client/server * get the subsecond cli thing working again * back to list of builds but with accessors * migrate some methods to workspace, clean up impl a bit * pass mode through build context * use a build context, similar to cargo/rustc * merge BuildUpdate and HandleUpdate into BuilderUpdate * Move more resolution of args into runner * migrate some methods to runner * hoist out fullstack * fix build request resolver * yay request is cleaned up * fixup resolution a bit * spawn off the build! * re-wire android autodetection * re-wire android autodetection * wire back up a few more things * re-wire tooling verification * I think it's mostly in the right condition for regular app * add depinfo parser * yay okay works with regular apps again * full rebuilds are back * rewire file wtcher * wire patch! * yayyyyyy patching works and is really fast * yay android works * clean up stdout piping * yayyy memap on android so we don't need root * create a static wasm jump table * wip: global offset table for wasm * wip... thinking about making the patches relocatable * wip: customized relocation * YESSSSSS RELOCATABLE WASM * clean up impl a bit * lil bit more cleanup * lil bit more * lil bit more cleanup * lil bit more cleanup, support 32 bit platforms * sick, wasm is completely pic * hmmmm not quite working yet * woooo, patches loading from rust+wasm * integrated into the cli 😎 * condense a bit * cleaned up request a bit * bust fingerprints * Make the file diffing logic a bit more sensible * still working through a few issues, server launches but is acting weird * fix merge conflict * more merge issues * remove stuff we don't want anymore * revert name change * wip: server not working anymore :( * split apart "dioxus fullstack" into "dioxus server" and "dioxus web" * fixup a few more compile errors * grrr opening server * simultaneous frontend and backend patching * use rustc wrapper * wip: passing assets to jumptable * migrate project-like examples * patchy patchy server fns! * rollback some random changes * unwind the js files * rollback hash * no need for patch * more cleanups * lil bit more cleanup, remove some old cruft * allow patching when wasm webpage is closed * tiny bit more robust * lil bit of clean up * clean up ws code * bit more clean ups * condense * undo file moves * move back other project * move other project * migrate out harness and janky CLI * fix compile * anonymize some files, heavily document others * lots more documentation, for posterity! * ton more docs * clean up the cli a bit more * more ws cleanup * more cleanup * migrate build id to cli-config * add command file handling * random small cleanups * wip.... * wip.... * fix: use workspace for krate discovery * fix panic logging on serve by placing it *after* the logs * reorder logging to be sensible on success * bring back swc * ws cruft * small patches to relocs in loader * bump krates * hoist main funcs into the ifunc table, fixing wasm * fix tui output for long lines * add more linker logging * wow, incredible, we build fat binaries, holy heckkkkk * fix args for frontend/backend * small cleanups * properly send along build IDs to clients * fix workspace compiles * clean up jump table work a bit * clean up logging a bit * open existing * fixup some commands * open existing browser integration * wire up the new dx serve client/server syntax * fixup `dx run` command * bring back some old functionality * fix serverfn feature sets * remove toast animation and simplify its code a bit * less intrusive toast * dont change build status when patching on web * add proper cache busting to vcomponent * clean up the patch file a bit * more lints/checks cleaned up * go back to TargetArgs but a BuildTargets resolved struct * clean up more nits * use an atomicptr for jumptable * fix interaction with initial web load + suspense * don't run asset system on wasm/bindgen js in dev * reduce blast radius of PR * use profile to determine if we're in release * cleanup prod html template * cleanup profiles * fix feature resolution of fullstack * if fullstack is explicitly false, don't use it * light cleanups * drop carg config2 * pass along linker args properly * make workspace examples compile with dx again * fewer unwraps and better error loggign * bit more error handlign * small cleanup * drive-by cleanups * use queries instead of initialize for aslr stuff * fix hotpatch with higher opt level * fix aslr bug from multiple clients * fix merge conflict * fix typos * clippy * fix miscompile * fix doctest * properly rollback wry to 0.45 * fix markdown path issues, other test issues * fix test in router * fix release mode test * fix some more tests, clean up a few more items * use fnptr instead of typeid * clean up memozation via subsecond in core * use "main" as sentinel * fix imports and re-exports * get off __rust_alloc for aslr offset detection * wip * wip... fixing unoptimized web * hmmmm mmmm * close,ish, still missing the wasmbindgen dynaimc imports * aha! full wasm works now - disable externref when compilng patch modules - manually link wasm instrinsics - displace the __wbindgen_placeholder__ table with the synthesized wbg imports what an awful pain. There might be a few items missing, so will test against the docsite shortly * remove failing test * hmmmm things aren't perfect * from what I can tell, relocatable wasm just doesn't work with wasm-bindgen sad https://github.com/rustwasm/wasm-bindgen/issues/1420 * AI IS LITERALLY THE WORST * preserve wasm snippets across compiles * IT HOTPATCHES THE DOCSITE, WE ARE LIVING IN THE FUTURE * remove the macos cfg-out * fix: need to allocate 1 more page * fix url issue * properly span props * properly fill in extra symbols * light cleanups and good docs * delete cruft, better docs * More cleanups and simplifications * clippy, typos * remove harnesses * wasm-bindgen error logging * clean up deps * remove cruft * proper priv/pub * use ifuncs for all env imports * cruft * use par_iter, better output * implement a cache for the base module * delete custom sections * cache the main native module too * add par iter in a few more places * clippy, map ident * better logging and status bars * wip: some small windows support * wip more windowss * fix weird regression * implement windows jump table * small fixes for windows * better windows support * windows support pt2 * rust lld is not msvc * whole archive the rlibs * ignore windows and sys crates * logging * ohhhhhh, pop out dlls * dedupe * thANK GOD FAT LINKING WORKS ON WINDOWS * hoist caching * implement patch based on cache * fix location for pdb * pass data symbols, correct object * add all symbol for windows * add log * treat none rva as undefined * whoops * handle windows calling convention * dont use relocated arm64 stub windows * not all are text? * WINDOWS FUNCTION PATCHING WORKS but issue with statics? * use our own entry * handle windows imp symbols * add log * jump to the real symbol not the layout * whoops * hotpatching windows works completely holy heck yay * drop linker args crate * disable ssg playwright for now * light cleanups * some cleanups / preps: - bump to serverfn 0.8.2 - fix playwright test - fix clippy * use fix from #4042 * bring in other fix for liveview * hoist some configuration * use __aslr_reference instead of aslr_reference * add better errors to patcherror * fix android * panic if patching fails * clean up tomls * wip: fixing wasm compile for server * fullstack wasm again * fix cfg * fixup feature detection * fix issue with features * fix playwright! * add barebones template test harness * fix compatibility between the two hotreload engines * fix preldue of server * fix check for wasm-opt * small fixes
This commit is contained in:
Jonathan Kelley authored and GitHub committed
1 parent
71d85e6dd5
commit
d976ca184a
142 files changed
+17982
-7156
No files matched your search
@@ -0,0 +1,384 @@
|
||||
use crate::CliSettings;
|
||||
use crate::Result;
|
||||
use crate::{config::DioxusConfig, AndroidTools};
|
||||
use anyhow::Context;
|
||||
use ignore::gitignore::Gitignore;
|
||||
use krates::KrateDetails;
|
||||
use krates::{Cmd, Krates, NodeId};
|
||||
use std::path::Path;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use target_lexicon::Triple;
|
||||
use tokio::process::Command;
|
||||
|
||||
pub struct Workspace {
|
||||
pub(crate) krates: Krates,
|
||||
pub(crate) settings: CliSettings,
|
||||
pub(crate) wasm_opt: Option<PathBuf>,
|
||||
pub(crate) sysroot: PathBuf,
|
||||
pub(crate) rustc_version: String,
|
||||
pub(crate) ignore: Gitignore,
|
||||
pub(crate) cargo_toml: cargo_toml::Manifest,
|
||||
pub(crate) android_tools: Option<Arc<AndroidTools>>,
|
||||
}
|
||||
|
||||
impl Workspace {
|
||||
/// Load the workspace from the current directory. This is cached and will only be loaded once.
|
||||
pub async fn current() -> Result<Arc<Workspace>> {
|
||||
static WS: tokio::sync::Mutex<Option<Arc<Workspace>>> = tokio::sync::Mutex::const_new(None);
|
||||
|
||||
// Lock the workspace to prevent multiple threads from loading it at the same time
|
||||
// If loading the workspace failed the first time, it won't be set and therefore permeate an error.
|
||||
let mut lock = WS.lock().await;
|
||||
if let Some(ws) = lock.as_ref() {
|
||||
return Ok(ws.clone());
|
||||
}
|
||||
|
||||
tracing::debug!("Loading workspace!");
|
||||
|
||||
let cmd = Cmd::new();
|
||||
let mut builder = krates::Builder::new();
|
||||
builder.workspace(true);
|
||||
let krates = builder
|
||||
.build(cmd, |_| {})
|
||||
.context("Failed to run cargo metadata")?;
|
||||
|
||||
let settings = CliSettings::global_or_default();
|
||||
let sysroot = Command::new("rustc")
|
||||
.args(["--print", "sysroot"])
|
||||
.output()
|
||||
.await
|
||||
.map(|out| String::from_utf8(out.stdout))?
|
||||
.context("Failed to extract rustc sysroot output")?;
|
||||
|
||||
let rustc_version = Command::new("rustc")
|
||||
.args(["--version"])
|
||||
.output()
|
||||
.await
|
||||
.map(|out| String::from_utf8(out.stdout))?
|
||||
.context("Failed to extract rustc version output")?;
|
||||
|
||||
let wasm_opt = which::which("wasm-opt").ok();
|
||||
|
||||
let ignore = Self::workspace_gitignore(krates.workspace_root().as_std_path());
|
||||
|
||||
let cargo_toml =
|
||||
cargo_toml::Manifest::from_path(krates.workspace_root().join("Cargo.toml"))
|
||||
.context("Failed to load Cargo.toml")?;
|
||||
|
||||
let android_tools = crate::build::get_android_tools();
|
||||
|
||||
let workspace = Arc::new(Self {
|
||||
krates,
|
||||
settings,
|
||||
wasm_opt,
|
||||
sysroot: sysroot.trim().into(),
|
||||
rustc_version: rustc_version.trim().into(),
|
||||
ignore,
|
||||
cargo_toml,
|
||||
android_tools,
|
||||
});
|
||||
|
||||
lock.replace(workspace.clone());
|
||||
|
||||
Ok(workspace)
|
||||
}
|
||||
|
||||
pub fn android_tools(&self) -> Result<Arc<AndroidTools>> {
|
||||
Ok(self
|
||||
.android_tools
|
||||
.clone()
|
||||
.context("Android not installed properly. Please set the `ANDROID_NDK_HOME` environment variable to the root of your NDK installation.")?)
|
||||
}
|
||||
|
||||
pub fn is_release_profile(&self, profile: &str) -> bool {
|
||||
// Check if the profile inherits from release by traversing the `inherits` chain
|
||||
let mut current_profile_name = profile;
|
||||
|
||||
// Try to find the current profile in the custom profiles section
|
||||
while let Some(profile_settings) = self.cargo_toml.profile.custom.get(current_profile_name)
|
||||
{
|
||||
if profile == "release" {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Check what this profile inherits from
|
||||
match &profile_settings.inherits {
|
||||
// Otherwise, continue checking the profile it inherits from
|
||||
Some(inherits_name) => current_profile_name = inherits_name,
|
||||
|
||||
// This profile doesn't explicitly inherit anything, so the chain ends here.
|
||||
// Since it didn't lead to "release", return false.
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
|
||||
false
|
||||
}
|
||||
|
||||
#[allow(unused)]
|
||||
pub fn rust_lld(&self) -> PathBuf {
|
||||
self.sysroot
|
||||
.join("lib")
|
||||
.join("rustlib")
|
||||
.join(Triple::host().to_string())
|
||||
.join("bin")
|
||||
.join("rust-lld")
|
||||
}
|
||||
|
||||
/// Return the path to the `cc` compiler
|
||||
///
|
||||
/// This is used for the patching system to run the linker.
|
||||
/// We could also just use lld given to us by rust itself.
|
||||
pub fn cc(&self) -> PathBuf {
|
||||
PathBuf::from("cc")
|
||||
}
|
||||
|
||||
/// The windows linker
|
||||
pub fn lld_link(&self) -> PathBuf {
|
||||
self.gcc_ld_dir().join("lld-link")
|
||||
}
|
||||
|
||||
pub fn wasm_ld(&self) -> PathBuf {
|
||||
self.gcc_ld_dir().join("wasm-ld")
|
||||
}
|
||||
|
||||
// wasm-ld: ./rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/wasm-ld
|
||||
// rust-lld: ./rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rust-lld
|
||||
fn gcc_ld_dir(&self) -> PathBuf {
|
||||
self.sysroot
|
||||
.join("lib")
|
||||
.join("rustlib")
|
||||
.join(Triple::host().to_string())
|
||||
.join("bin")
|
||||
.join("gcc-ld")
|
||||
}
|
||||
|
||||
pub fn has_wasm32_unknown_unknown(&self) -> bool {
|
||||
self.sysroot
|
||||
.join("lib/rustlib/wasm32-unknown-unknown")
|
||||
.exists()
|
||||
}
|
||||
|
||||
/// Find the "main" package in the workspace. There might not be one!
|
||||
pub fn find_main_package(&self, package: Option<String>) -> Result<NodeId> {
|
||||
if let Some(package) = package {
|
||||
let mut workspace_members = self.krates.workspace_members();
|
||||
let found = workspace_members.find_map(|node| {
|
||||
if let krates::Node::Krate { id, krate, .. } = node {
|
||||
if krate.name == package {
|
||||
return Some(id);
|
||||
}
|
||||
}
|
||||
None
|
||||
});
|
||||
|
||||
if found.is_none() {
|
||||
tracing::error!("Could not find package {package} in the workspace. Did you forget to add it to the workspace?");
|
||||
tracing::error!("Packages in the workspace:");
|
||||
for package in self.krates.workspace_members() {
|
||||
if let krates::Node::Krate { krate, .. } = package {
|
||||
tracing::error!("{}", krate.name());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let kid = found.ok_or_else(|| anyhow::anyhow!("Failed to find package {package}"))?;
|
||||
|
||||
return Ok(self.krates.nid_for_kid(kid).unwrap());
|
||||
};
|
||||
|
||||
// Otherwise find the package that is the closest parent of the current directory
|
||||
let current_dir = std::env::current_dir()?;
|
||||
let current_dir = current_dir.as_path();
|
||||
|
||||
// Go through each member and find the path that is a parent of the current directory
|
||||
let mut closest_parent = None;
|
||||
for member in self.krates.workspace_members() {
|
||||
if let krates::Node::Krate { id, krate, .. } = member {
|
||||
let member_path = krate.manifest_path.parent().unwrap();
|
||||
if let Ok(path) = current_dir.strip_prefix(member_path.as_std_path()) {
|
||||
let len = path.components().count();
|
||||
match closest_parent {
|
||||
Some((_, closest_parent_len)) => {
|
||||
if len < closest_parent_len {
|
||||
closest_parent = Some((id, len));
|
||||
}
|
||||
}
|
||||
None => {
|
||||
closest_parent = Some((id, len));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let kid = closest_parent
|
||||
.map(|(id, _)| id)
|
||||
.with_context(|| {
|
||||
let bin_targets = self.krates.workspace_members().filter_map(|krate|match krate {
|
||||
krates::Node::Krate { krate, .. } if krate.targets.iter().any(|t| t.kind.contains(&krates::cm::TargetKind::Bin))=> {
|
||||
Some(format!("- {}", krate.name))
|
||||
}
|
||||
_ => None
|
||||
}).collect::<Vec<_>>();
|
||||
format!("Failed to find binary package to build.\nYou need to either run dx from inside a binary crate or specify a binary package to build with the `--package` flag. Try building again with one of the binary packages in the workspace:\n{}", bin_targets.join("\n"))
|
||||
})?;
|
||||
|
||||
let package = self.krates.nid_for_kid(kid).unwrap();
|
||||
Ok(package)
|
||||
}
|
||||
|
||||
pub fn load_dioxus_config(&self, package: NodeId) -> Result<Option<DioxusConfig>> {
|
||||
// Walk up from the cargo.toml to the root of the workspace looking for Dioxus.toml
|
||||
let mut current_dir = self.krates[package]
|
||||
.manifest_path
|
||||
.parent()
|
||||
.unwrap()
|
||||
.as_std_path()
|
||||
.to_path_buf()
|
||||
.canonicalize()?;
|
||||
|
||||
let workspace_path = self
|
||||
.krates
|
||||
.workspace_root()
|
||||
.as_std_path()
|
||||
.to_path_buf()
|
||||
.canonicalize()?;
|
||||
|
||||
let mut dioxus_conf_file = None;
|
||||
while current_dir.starts_with(&workspace_path) {
|
||||
let config = ["Dioxus.toml", "dioxus.toml"]
|
||||
.into_iter()
|
||||
.map(|file| current_dir.join(file))
|
||||
.find(|path| path.is_file());
|
||||
|
||||
// Try to find Dioxus.toml in the current directory
|
||||
if let Some(new_config) = config {
|
||||
dioxus_conf_file = Some(new_config.as_path().to_path_buf());
|
||||
break;
|
||||
}
|
||||
// If we can't find it, go up a directory
|
||||
current_dir = current_dir
|
||||
.parent()
|
||||
.context("Failed to find Dioxus.toml")?
|
||||
.to_path_buf();
|
||||
}
|
||||
|
||||
let Some(dioxus_conf_file) = dioxus_conf_file else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
toml::from_str::<DioxusConfig>(&std::fs::read_to_string(&dioxus_conf_file)?)
|
||||
.map_err(|err| {
|
||||
anyhow::anyhow!("Failed to parse Dioxus.toml at {dioxus_conf_file:?}: {err}").into()
|
||||
})
|
||||
.map(Some)
|
||||
}
|
||||
|
||||
/// Create a new gitignore map for this target crate
|
||||
///
|
||||
/// todo(jon): this is a bit expensive to build, so maybe we should cache it?
|
||||
pub fn workspace_gitignore(workspace_dir: &Path) -> Gitignore {
|
||||
let mut ignore_builder = ignore::gitignore::GitignoreBuilder::new(workspace_dir);
|
||||
ignore_builder.add(workspace_dir.join(".gitignore"));
|
||||
|
||||
// todo!()
|
||||
// let workspace_dir = self.workspace_dir();
|
||||
// ignore_builder.add(workspace_dir.join(".gitignore"));
|
||||
|
||||
for path in Self::default_ignore_list() {
|
||||
ignore_builder
|
||||
.add_line(None, path)
|
||||
.expect("failed to add path to file excluded");
|
||||
}
|
||||
|
||||
ignore_builder.build().unwrap()
|
||||
}
|
||||
|
||||
pub fn ignore_for_krate(&self, path: &Path) -> ignore::gitignore::Gitignore {
|
||||
let mut ignore_builder = ignore::gitignore::GitignoreBuilder::new(path);
|
||||
for path in Self::default_ignore_list() {
|
||||
ignore_builder
|
||||
.add_line(None, path)
|
||||
.expect("failed to add path to file excluded");
|
||||
}
|
||||
ignore_builder.build().unwrap()
|
||||
}
|
||||
|
||||
pub fn default_ignore_list() -> Vec<&'static str> {
|
||||
vec![
|
||||
".git",
|
||||
".github",
|
||||
".vscode",
|
||||
"target",
|
||||
"node_modules",
|
||||
"dist",
|
||||
"*~",
|
||||
".*",
|
||||
"*.lock",
|
||||
"*.log",
|
||||
]
|
||||
}
|
||||
|
||||
pub(crate) fn workspace_root(&self) -> PathBuf {
|
||||
self.krates.workspace_root().as_std_path().to_path_buf()
|
||||
}
|
||||
|
||||
/// Returns the root of the crate that the command is run from, without calling `cargo metadata`
|
||||
///
|
||||
/// If the command is run from the workspace root, this will return the top-level Cargo.toml
|
||||
pub(crate) fn crate_root_from_path() -> Result<PathBuf> {
|
||||
/// How many parent folders are searched for a `Cargo.toml`
|
||||
const MAX_ANCESTORS: u32 = 10;
|
||||
|
||||
/// Checks if the directory contains `Cargo.toml`
|
||||
fn contains_manifest(path: &Path) -> bool {
|
||||
std::fs::read_dir(path)
|
||||
.map(|entries| {
|
||||
entries
|
||||
.filter_map(Result::ok)
|
||||
.any(|ent| &ent.file_name() == "Cargo.toml")
|
||||
})
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
// From the current directory we work our way up, looking for `Cargo.toml`
|
||||
std::env::current_dir()
|
||||
.ok()
|
||||
.and_then(|mut wd| {
|
||||
for _ in 0..MAX_ANCESTORS {
|
||||
if contains_manifest(&wd) {
|
||||
return Some(wd);
|
||||
}
|
||||
if !wd.pop() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
None
|
||||
})
|
||||
.ok_or_else(|| {
|
||||
crate::Error::Cargo("Failed to find directory containing Cargo.toml".to_string())
|
||||
})
|
||||
}
|
||||
|
||||
/// Returns the properly canonicalized path to the dx executable, used for linking and wrapping rustc
|
||||
pub(crate) fn path_to_dx() -> Result<PathBuf> {
|
||||
Ok(
|
||||
dunce::canonicalize(std::env::current_exe().context("Failed to find dx")?)
|
||||
.context("Failed to find dx")?,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for Workspace {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("Workspace")
|
||||
.field("krates", &"..")
|
||||
.field("settings", &self.settings)
|
||||
.field("rustc_version", &self.rustc_version)
|
||||
.field("sysroot", &self.sysroot)
|
||||
.field("wasm_opt", &self.wasm_opt)
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user