Files
dioxus/packages/cli/src/cli/link.rs
T
Jonathan Kelley d976ca184a Binary patching rust hot-reloading, sub-second rebuilds, independent server/client hot-reload (#3797)
Enable runtime rust hot patching.

* roll back wry to 0.45

* cleanup rustc

* wip

* wip: fix client/server

* get the subsecond cli thing working again

* back to list of builds but with accessors

* migrate some methods to workspace, clean up impl a bit

* pass mode through build context

* use a build context, similar to cargo/rustc

* merge BuildUpdate and HandleUpdate into BuilderUpdate

* Move more resolution of args into runner

* migrate some methods to runner

* hoist out fullstack

* fix build request resolver

* yay request is cleaned up

* fixup resolution a bit

* spawn off the build!

* re-wire android autodetection

* re-wire android autodetection

* wire back up a few more things

* re-wire tooling verification

* I think it's mostly in the right condition for regular app

* add depinfo parser

* yay okay works with regular apps again

* full rebuilds are back

* rewire file wtcher

* wire patch!

* yayyyyyy patching works and is really fast

* yay android works

* clean up stdout piping

* yayyy memap on android so we don't need root

* create a static wasm jump table

* wip: global offset table for wasm

* wip... thinking about making the patches relocatable

* wip: customized relocation

* YESSSSSS RELOCATABLE WASM

* clean up impl a bit

* lil bit more cleanup

* lil bit more

* lil bit more cleanup

* lil bit more cleanup, support 32 bit platforms

* sick, wasm is completely pic

* hmmmm not quite working yet

* woooo, patches loading from rust+wasm

* integrated into the cli 😎

* condense a bit

* cleaned up request a bit

* bust fingerprints

* Make the file diffing logic a bit more sensible

* still working through a few issues, server launches but is acting weird

* fix merge conflict

* more merge issues

* remove stuff we don't want anymore

* revert name change

* wip: server not working anymore :(

* split apart "dioxus fullstack" into "dioxus server" and "dioxus web"

* fixup a few more compile errors

* grrr opening server

* simultaneous frontend and backend patching

* use rustc wrapper

* wip: passing assets to jumptable

* migrate project-like examples

* patchy  patchy server fns!

* rollback some random changes

* unwind the js files

* rollback hash

* no need for patch

* more cleanups

* lil bit more cleanup, remove some old cruft

* allow patching when wasm webpage is closed

* tiny bit more robust

* lil bit of clean up

* clean up ws code

* bit more clean ups

* condense

* undo file moves

* move back other project

* move other project

* migrate out harness and janky CLI

* fix compile

* anonymize some files, heavily document others

* lots more documentation, for posterity!

* ton more docs

* clean up the cli a bit more

* more ws cleanup

* more cleanup

* migrate build id to cli-config

* add command file handling

* random small cleanups

* wip....

* wip....

* fix: use workspace for krate discovery

* fix panic logging on serve by placing it *after* the logs

* reorder logging to be sensible on success

* bring back swc

* ws cruft

* small patches to relocs in loader

* bump krates

* hoist main funcs into the ifunc table, fixing wasm

* fix tui output for long lines

* add more linker logging

* wow, incredible, we build fat binaries, holy heckkkkk

* fix args for frontend/backend

* small cleanups

* properly send along build IDs to clients

* fix workspace compiles

* clean up jump table work a bit

* clean up logging a bit

* open existing

* fixup some commands

* open existing browser integration

* wire up the new dx serve client/server syntax

* fixup `dx run` command

* bring back some old functionality

* fix serverfn feature sets

* remove toast animation and simplify its code a bit

* less intrusive toast

* dont change build status when patching on web

* add proper cache busting to vcomponent

* clean up the patch file a bit

* more lints/checks cleaned up

* go back to TargetArgs but a BuildTargets resolved struct

* clean up more nits

* use an atomicptr for jumptable

* fix interaction with initial web load + suspense

* don't run asset system on wasm/bindgen js in dev

* reduce blast radius of PR

* use profile to determine if we're in release

* cleanup prod html template

* cleanup profiles

* fix feature resolution of fullstack

* if fullstack is explicitly false, don't use it

* light cleanups

* drop carg config2

* pass along linker args properly

* make workspace examples compile with dx again

* fewer unwraps and better error loggign

* bit more error handlign

* small cleanup

* drive-by cleanups

* use queries instead of initialize for aslr stuff

* fix hotpatch with higher opt level

* fix aslr bug from multiple clients

* fix merge conflict

* fix typos

* clippy

* fix miscompile

* fix doctest

* properly rollback wry to 0.45

* fix markdown path issues, other test issues

* fix test in router

* fix release mode test

* fix some more tests, clean up a few more items

* use fnptr instead of typeid

* clean up memozation via subsecond in core

* use "main" as sentinel

* fix imports and re-exports

* get off __rust_alloc for aslr offset detection

* wip

* wip... fixing unoptimized web

* hmmmm mmmm

* close,ish, still missing the wasmbindgen dynaimc imports

* aha! full wasm works now

- disable externref when compilng patch modules
- manually link wasm instrinsics
- displace the __wbindgen_placeholder__ table with the synthesized wbg imports

what an awful pain.

There might be a few items missing, so will test against the docsite shortly

* remove failing test

* hmmmm things aren't perfect

* from what I can tell, relocatable wasm just doesn't work with wasm-bindgen

sad

https://github.com/rustwasm/wasm-bindgen/issues/1420

* AI IS LITERALLY THE WORST

* preserve wasm snippets across compiles

* IT HOTPATCHES THE DOCSITE, WE ARE LIVING IN THE FUTURE

* remove the macos cfg-out

* fix: need to allocate 1 more page

* fix url issue

* properly span props

* properly fill in extra symbols

* light cleanups and good docs

* delete cruft, better docs

* More cleanups and simplifications

* clippy, typos

* remove harnesses

* wasm-bindgen error logging

* clean up deps

* remove cruft

* proper priv/pub

* use ifuncs for all env imports

* cruft

* use par_iter, better output

* implement a cache for the base module

* delete custom sections

* cache the main native module too

* add par iter in a few more places

* clippy, map ident

* better logging and status bars

* wip: some small windows support

* wip more windowss

* fix weird regression

* implement windows jump table

* small fixes for windows

* better windows support

* windows support pt2

* rust lld is not msvc

* whole archive the rlibs

* ignore windows and sys crates

* logging

* ohhhhhh, pop out dlls

* dedupe

* thANK GOD FAT LINKING WORKS ON WINDOWS

* hoist caching

* implement patch based on cache

* fix location for pdb

* pass data symbols, correct object

* add all symbol for windows

* add log

* treat none rva as undefined

* whoops

* handle windows calling convention

* dont use relocated arm64 stub windows

* not all are text?

* WINDOWS FUNCTION PATCHING WORKS

but issue with statics?

* use our own entry

* handle windows imp symbols

* add log

* jump to the real symbol not the layout

* whoops

* hotpatching windows works completely holy heck yay

* drop linker args crate

* disable ssg playwright for now

* light cleanups

* some cleanups / preps:
- bump to serverfn 0.8.2
- fix playwright test
- fix clippy

* use fix from #4042

* bring in other fix for liveview

* hoist some configuration

* use __aslr_reference instead of aslr_reference

* add better errors to patcherror

* fix android

* panic if patching fails

* clean up tomls

* wip: fixing wasm compile for server

* fullstack wasm again

* fix cfg

* fixup feature detection

* fix issue with features

* fix playwright!

* add barebones template test harness

* fix compatibility between the two hotreload engines

* fix preldue of server

* fix check for wasm-opt

* small fixes
2025-05-06 23:08:59 -07:00

253 lines
11 KiB
Rust

use crate::Result;
use anyhow::Context;
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use target_lexicon::Triple;
/// `dx` can act as a linker in a few scenarios. Note that we don't *actually* implement the linker logic,
/// instead just proxying to a specified linker (or not linking at all!).
///
/// This comes in two flavors:
/// --------------------------
/// - `BaseLink`: We are linking dependencies and want to dynamically select the linker from the environment.
/// This is mostly implemented for Android where the linker is selected in part by the
/// device connected over ADB which can not be determined by .cargo/Config.toml.
/// We implemented this because previous setups like cargo mobile required a hard-coded
/// linker path in your project which does not work in team-based setups.
///
/// - `NoLink`: We are not linking at all, and instead deferring our linking to the driving process,
/// usually being `dx` itself. In this case, we are just writing the linker args to a file
/// and then outputting a dummy object file to satisfy the linker. This is generally used
/// by the binary patching engine since we need to actually do "real linker logic" like
/// traversing object files and satisfying missing symbols. That process is *much* easier
/// to do in the driving host process when we have all the information available. Unfortunately,
/// rustc doesn't provide a "real" way of granularly stepping through the compile process
/// so this is basically a hack.
///
/// We use "BaseLink" when a linker is specified, and "NoLink" when it is not. Both generate a resulting
/// object file.
#[derive(Debug)]
pub struct LinkAction {
pub linker: Option<PathBuf>,
pub triple: Triple,
pub link_args_file: PathBuf,
pub link_err_file: PathBuf,
}
/// The linker flavor to use. This influences the argument style that gets passed to the linker.
/// We're imitating the rustc linker flavors here.
///
/// https://doc.rust-lang.org/beta/nightly-rustc/rustc_target/spec/enum.LinkerFlavor.html
#[derive(Debug, Serialize, Deserialize, PartialEq, Clone)]
pub enum LinkerFlavor {
Gnu,
Darwin,
WasmLld,
Unix,
Msvc,
}
impl LinkAction {
const DX_LINK_ARG: &str = "DX_LINK";
const DX_ARGS_FILE: &str = "DX_LINK_ARGS_FILE";
const DX_ERR_FILE: &str = "DX_LINK_ERR_FILE";
const DX_LINK_TRIPLE: &str = "DX_LINK_TRIPLE";
const DX_LINK_CUSTOM_LINKER: &str = "DX_LINK_CUSTOM_LINKER";
/// Should we write the input arguments to a file (aka act as a linker subprocess)?
///
/// Just check if the magic env var is set
pub(crate) fn from_env() -> Option<Self> {
if std::env::var(Self::DX_LINK_ARG).is_err() {
return None;
}
Some(Self {
linker: std::env::var(Self::DX_LINK_CUSTOM_LINKER)
.ok()
.map(PathBuf::from),
link_args_file: std::env::var(Self::DX_ARGS_FILE)
.expect("Linker args file not set")
.into(),
link_err_file: std::env::var(Self::DX_ERR_FILE)
.expect("Linker error file not set")
.into(),
triple: std::env::var(Self::DX_LINK_TRIPLE)
.expect("Linker triple not set")
.parse()
.expect("Failed to parse linker triple"),
})
}
pub(crate) fn write_env_vars(&self, env_vars: &mut Vec<(&str, String)>) -> Result<()> {
env_vars.push((Self::DX_LINK_ARG, "1".to_string()));
env_vars.push((
Self::DX_ARGS_FILE,
dunce::canonicalize(&self.link_args_file)?
.to_string_lossy()
.to_string(),
));
env_vars.push((
Self::DX_ERR_FILE,
dunce::canonicalize(&self.link_err_file)?
.to_string_lossy()
.to_string(),
));
env_vars.push((Self::DX_LINK_TRIPLE, self.triple.to_string()));
if let Some(linker) = &self.linker {
env_vars.push((
Self::DX_LINK_CUSTOM_LINKER,
dunce::canonicalize(linker)?.to_string_lossy().to_string(),
));
}
Ok(())
}
pub(crate) async fn run_link(self) {
let link_err_file = self.link_err_file.clone();
let res = self.run_link_inner().await;
if let Err(err) = res {
// If we failed to run the linker, we need to write the error to the file
// so that the main process can read it.
_ = std::fs::create_dir_all(link_err_file.parent().unwrap());
_ = std::fs::write(link_err_file, format!("Linker error: {err}"));
}
}
/// Write the incoming linker args to a file
///
/// The file will be given by the dx-magic-link-arg env var itself, so we use
/// it both for determining if we should act as a linker and the for the file name itself.
async fn run_link_inner(self) -> Result<()> {
let mut args: Vec<_> = std::env::args().collect();
if args.is_empty() {
return Ok(());
}
handle_linker_command_file(&mut args);
// Write the linker args to a file for the main process to read
// todo: we might need to encode these as escaped shell words in case newlines are passed
std::fs::write(self.link_args_file, args.join("\n"))?;
// If there's a linker specified, we use that. Otherwise, we write a dummy object file to satisfy
// any post-processing steps that rustc does.
match self.linker {
Some(linker) => {
let res = std::process::Command::new(linker)
.args(args.iter().skip(1))
.output()
.expect("Failed to run linker");
if !res.stderr.is_empty() || !res.stdout.is_empty() {
_ = std::fs::create_dir_all(self.link_err_file.parent().unwrap());
_ = std::fs::write(
self.link_err_file,
format!(
"Linker error: {}\n{}",
String::from_utf8_lossy(&res.stdout),
String::from_utf8_lossy(&res.stderr)
),
);
}
}
None => {
// Extract the out path - we're going to write a dummy object file to satisfy the linker
let out_file: PathBuf = match self.triple.operating_system {
target_lexicon::OperatingSystem::Windows => {
let out_arg = args.iter().find(|arg| arg.starts_with("/OUT")).unwrap();
out_arg.trim_start_matches("/OUT:").to_string().into()
}
_ => {
let out = args.iter().position(|arg| arg == "-o").unwrap();
args[out + 1].clone().into()
}
};
// This creates an object file that satisfies rust's use of llvm-objcopy
//
// I'd rather we *not* do this and instead generate a truly linked file (and then delete it) but
// this at least lets us delay linking until the host compiler is ready.
//
// This is because our host compiler is a stateful server and not a stateless linker.
//
// todo(jon): do we use Triple::host or the target triple? I think I ran into issues
// using the target triple, hence the use of "host" but it might not even matter?
let triple = Triple::host();
let format = match triple.binary_format {
target_lexicon::BinaryFormat::Elf => object::BinaryFormat::Elf,
target_lexicon::BinaryFormat::Coff => object::BinaryFormat::Coff,
target_lexicon::BinaryFormat::Macho => object::BinaryFormat::MachO,
target_lexicon::BinaryFormat::Wasm => object::BinaryFormat::Wasm,
target_lexicon::BinaryFormat::Xcoff => object::BinaryFormat::Xcoff,
target_lexicon::BinaryFormat::Unknown => todo!(),
_ => todo!("Binary format not supported"),
};
let arch = match triple.architecture {
target_lexicon::Architecture::Wasm32 => object::Architecture::Wasm32,
target_lexicon::Architecture::Wasm64 => object::Architecture::Wasm64,
target_lexicon::Architecture::X86_64 => object::Architecture::X86_64,
target_lexicon::Architecture::Arm(_) => object::Architecture::Arm,
target_lexicon::Architecture::Aarch64(_) => object::Architecture::Aarch64,
target_lexicon::Architecture::LoongArch64 => object::Architecture::LoongArch64,
target_lexicon::Architecture::Unknown => object::Architecture::Unknown,
_ => todo!("Architecture not supported"),
};
let endian = match triple.endianness() {
Ok(target_lexicon::Endianness::Little) => object::Endianness::Little,
Ok(target_lexicon::Endianness::Big) => object::Endianness::Big,
Err(_) => todo!("Endianness not supported"),
};
let bytes = object::write::Object::new(format, arch, endian)
.write()
.context("Failed to emit stub link file")?;
// Write a dummy object file to satisfy rust/linker since it'll run llvm-objcopy
// ... I wish it *didn't* do that but I can't tell how to disable the linker without
// using --emit=obj which is not exactly what we want since that will still pull in
// the dependencies.
std::fs::create_dir_all(out_file.parent().unwrap())?;
std::fs::write(out_file, bytes)?;
}
}
Ok(())
}
}
pub fn handle_linker_command_file(args: &mut Vec<String>) {
// Handle command files, usually a windows thing.
if let Some(command) = args.iter().find(|arg| arg.starts_with('@')).cloned() {
let path = command.trim().trim_start_matches('@');
let file_binary = std::fs::read(path).unwrap();
// This may be a utf-16le file. Let's try utf-8 first.
let content = String::from_utf8(file_binary.clone()).unwrap_or_else(|_| {
// Convert Vec<u8> to Vec<u16> to convert into a String
let binary_u16le: Vec<u16> = file_binary
.chunks_exact(2)
.map(|a| u16::from_le_bytes([a[0], a[1]]))
.collect();
String::from_utf16_lossy(&binary_u16le)
});
// Gather linker args, and reset the args to be just the linker args
*args = content
.lines()
.map(|line| {
let line_parsed = line.trim().to_string();
let line_parsed = line_parsed.trim_end_matches('"').to_string();
let line_parsed = line_parsed.trim_start_matches('"').to_string();
line_parsed
})
.collect();
}
}